
Volume 2 v 



Selected Papers on the Practice of Educational 
Communications and Technology Presented at the 2007 
Annual Convention of the Association for Educational 
Communications and Technology 




Sponsored by the Research and Theory Division 
Anaheim, CA 



Editor: Michael Simonson 



Nova Southeastern University, North Miami Beach, Florida 



2007 Annual Proceedings - Anaheim: Volume #2 

Selected Papers 

On the Practice of Educational Communications and Technology 
Presented at 

The Annual Convention of the Association for Educational Communications and Technology 
Sponsored by the Research and Theory Division 

Anaheim, CA 

2007 



Editor: 

Michael Simonson 
Professor 

Instructional Technology and Distance Education 
Nova Southeastern University 
Fischler School of Education and Human Services 
North Miami Beach, FL 




Previous Proceedings Published in ERIC 



Year 


Location 


ED Number 


1979 


New Orleans 


171329 


1980 


Denver 


194061 


1981 


Philadelphia 


207487 


1982 


Dallas 


223191 -223326 


1983 


New Orleans 


231337 


1984 


Dallas 


243411 


1985 


Anaheim 


256301 


1986 


Las Vegas 


267753 


1987 


Atlanta 


285518 


1988 


New Orleans 


295621 


1989 


Dallas 


308805 


1990 


Anaheim 


323912 


1991 


Orlando 


334969 


1992 


Washington, D.C. 


347970-348041 


1993 


New Orleans 


362144 


1994 


Nashville 


373774 


1995 


Anaheim 


383284 


1996 


Indianapolis 


397772 


1997 


Albuquerque 


409832 


1998 


St. Louis 


423819 


1999 


Houston 


436128 


2000 


Long Beach 


444595 


2000 


Denver 


455756 


2001 


Atlanta 


470066 


2002 


Dallas 


Submitted to ERIC 


2003 


Anaheim 


Submitted to ERIC 


2004 


Chicago 


Submitted to ERIC 


2005 


Orlando 


Submitted to ERIC 


2006 


Dallas 


Submitted to ERIC 




Preface 



For the thirtieth year, the Research and Theory Division of the Association for Educational 
Communications and Technology (AECT) is sponsoring the publication of these Proceedings . This is 
Volume #2 of the 30' 1 ' Annual Proceedings of Selected Papers On the Practice of Educational 
Communications and Technology’. This volume includes papers presented at the national convention of the 
Association for Educational Communications and Technology held in Anaheim, CA. Copies were are 
available online at AECT.ORG. Volumes 1 and 2 are also available through the Educational Resources 
Clearinghouse (ERIC) system. 

This volume contains papers primarily dealing with instruction and training issues. Papers dealing with 
research and development are contained in the companion volume (Volume #1). 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings . The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 

Michael R. Simonson 
Editor 
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Business Process Experiential Simulation in an Engineering Environment 



Kim A. Armstrong, Ed.D. 

The Boeing Company 

Daniel J. Campbell, 

Vangent Inc. 

The Need 

A February 2004 memo from the Undersecretary of Defense, which includes an explicit requirement to use 
a “robust systems engineering approach” (][ 1) on all projects and contracts associated with the military. The Boeing 
Company’s long-term strategy is tied very closely to a continuance and maintenance of Department of Defense 
(DoD) projects as well as commercial customers. Regardless of whether the customer is the DoD or a commercial 
venture, each of Boeing’s customers expects a product delivered on time, within costs, and performing to the desired 
outcome. If any one of these expectations is not met, the result is customer dissatisfaction. Systems Engineering 
(SE) influences cost, schedule, and performance, so a solid SE approach on each program increases the likelihood of 
meeting customer expectations, which is directly tied to Boeing’s long term strategy of customer continuance and 
maintenance. Additionally, SE directly impacts Boeing’s bottom line profitability. 

The Boeing Company defines Systems Engineering as an interdisciplinary collaborative approach to derive, 
evolve, and verify a life cycle balanced solution that satisfies customer expectations and meets public acceptance. 
This definition places Systems Engineering more in the category of a business process that touches many 
stakeholders, than an engineering discipline. While Systems Engineering had been a practice within Boeing for 
many years, it had not achieved a level of implementation relative to the role it plays within the organization. To 
achieve an objective of becoming a recognized industry leader in Systems Engineering applications. The Boeing 
Company determined that Systems Engineering practices needed to receive greater attention and focus within the 
organization. 



The Solution 

A crucial challenge to achieving the objective was to modify the efficiency and order mindset of the 
engineer learner audience from a narrow engineering focus, to an orientation of adaptability and openness within a 
larger system with a wide variety of stakeholders (Svaboda & Wahlen, 2004/2005). To meet this challenge, Boeing 
partnered with Vangent Inc. to develop a blended Systems Engineering Leadership Program (SELP) with the 
primary affective objective of having the learners recognize the value Systems Engineering brings to a project or 
program. As shown in Figure 1, the SELP incorporates both synchronous and asynchronous online components, 
instructor-led components, classroom computer-based simulations, and program effectiveness evaluation strategies. 
The program is delivered in five segments: 

1 . Segment 1 : Pre-course Discussion (2 hours) 

2. Segment 2: Introductory Web-based training (WBT) (1.5 hours) and Simulation pre-read (1.5 hours) 

3. Segment 3: Classroom Sessions (4 days blend of Instructor-led training (ILT) and Web-based (or CD- 
ROM-based) simulation 

4. Segment 4: Follow-up (1.25 hours WBT) 

5. Segment 5: Post-course Discussion (4 hours in-person or virtual discussion group) 

Six months after completing the curriculum, learners complete a Post Course Impact Assessment which consists of a 
behavioral survey instrument designed to measure the extent to which course participants have been able to transfer 
what they learned in the course to their actual job performance. 
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Figure 1. The Systems Engineering Leadership Program 
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Theoretical Framework 

Bloom’s taxonomy of cognition provides a useful framework for understanding the curriculum structure. The 
taxonomy can help make sense of this complexity by breaking down the objectives into six hieratical categories: 
Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation (Bloom, 1994). These six categories 
are arranged on level of difficulty. In other words, a learner who is able to perform at the higher levels of the 
taxonomy is demonstrating a more complex level of cognitive thinking. The design approach arranges the SELP 
curriculum so that the learner: 

1 . Builds basic knowledge in the early portion of the Introduction WBT 

2. Begins to comprehend how the various chunks of knowledge interrelate during the Introductory WBT case 
studies. 

3. Continues to gain knowledge and comprehension, and starts to apply the knowledge to a simulation in 
segment three. 

4. Has opportunities to analyze the benefits and consequences of his or her actions during simulation debriefs 
in segment three. 

5. Is encouraged to synthesis the learning experience by conducting an on-the-job SE project between 
segments three and five. 

6. Is provided opportunities to evaluate the impact of the learning experience with the post-course assessment 
(segment four), the post course discussion (segment five), and the post course behaviors impact assessment. 

This diverse blend of activities and opportunities for reflection and self assessment, facilitate the higher-order 
critical thinking skills at the end of the taxonomy that are critical for SE success. 

The center piece of this curriculum is a learning team-based Systems Engineering simulation of the entire 
product/program life-cycle, which is interspersed throughout a four-day classroom session. In a simulation, the 
learner takes responsibility for the decision-making, experiencing the positive and negative consequences of those 
decisions. This results in the learner gaining valuable problem solving and decision-making skills. The simulation 
reinforces key concepts taught in the classroom using scenario-based activities, which challenge participants to use 
Systems Engineering principles to solve real-life problems. This allows the learners to apply previously acquired 
knowledge and skills to control an environment that replicates the real world. 
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The theoretical framework behind this simulation approach was based on Kolb’s (1984) four-stage 
framework of experiential learning; which includes concrete experience, reflective observation, abstract 
conceptualization, and active experimentation. Experiential learning occurs when individuals participate completely 
in the learning process, control its nature and direction, and the learning activities are primarily based upon direct 
confrontation with practical, real-world problems; and learner self-evaluation is the principal method of assessing 
progress or success. 

Simulation Construct 

The construct of the simulation is a blend of computer-based courseware, learning team discussion and 
activities, and large group presentations. Prior to attending the four day classroom session, the learners were 
provided with a pre-read proposal document outlining a conceptual product development program that would be the 
subject of the simulation. The simulation itself consisted of four interactive CD-ROM modules; one for each product 
life cycle phase, and a set of PowerPoint templates on which learning teams document assigned activities in 
preparation for debriefing presentations at the end of each phase. The computer-based portions of each simulation 
phase provided virtual interactions with key stakeholders of a simulated development program; using emails, 
voicemails, simulated phone calls and phone conferences, and simulated Internet research resources. 

The simulated emails consist of text and narration describing the current contextual setting, and an email 
"Inbox" which can contain up to 10 messages. Learners are directed to click on the email message in the Inbox 
(Figure 2) to open the message window and read the message. The email messages can contain text, and optional 
attached files or links to simulated Internet resources. Figure 3 provides an example of an email containing a link to 
a simulated Internet resource. In this example the link take the learners to a fictitious company’s web site which is 
contained on the simulation CD. 



SELP Plan /Acquire Simulation Identify Risks & Opportunities 

Engine, Sensor, & Communications TRLs / MRLs 



Course Map | Audio Script | Resources | Help | Exit 

•S^tems Engineering Leadership Program 



RO 10726 
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Now that required capabilities - MIRs 
and Technical Performance Measures 
for those capabilities have been 
established for the BSD. the project 
team is working to identify potential 
solutions for meeting the required 
capabilities, determine Technical and 
Manufacturing Readiness Levels of the 
various options, and identify risks and 
opportunities associated with each. 
The Program Manager and 5EIT Lead 
have asked the IPT leads to research 
potential solutions for their particular 
area of responsibility. 

The Program Manager has received 
some email responses to his request. 



1 D 0 From 



Subject 



Mid Dote: Today 



O IBSD Propulsion IP BSD Engine Options 
O [BSD Avionics IPT BSD Sensor Package 
O [BSD Avionics IPT BSD Communications Package 



THU 10/1272006 7:47 AM 
THU 10/12/200610:52 AM 
THU 10/12/2006 4:11 PM 




Figure 2. Simulated Email Inbox 
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BSD Concept Recommendations 



From: 

To: 

Cc: 

Subject 



(BSD Propulsion IPT Lead] 

(BSD Program Manager]; [BSD SE Focal] 

BSD Engine Options 



Sent Fri 3/09/2007 9 36 AM 



I have identified a couple of potential engine options for the BSD. There is a small company ±_ 
in Connecticut. M5 Technologies, which is really on the cutting edge of new engine 
technologies. They are developing an engine, which I think will easily fit our airframe, and 
that should be able to keep the BSD in the air for 3 days. 

They are also testing an experimental technology which promises to double engine fuel 
efficiency. They claim this technology will be ready in about a year. If their claims are 
accurate, then with just a bit of redesign we should be able to add enough fuel to reach the 
7-day requirement. 

Here is the link to the company website: 

www M5Tech.com 




36 AM 



Figure 3. Email Containing a Simulated Internet Resource 



At certain points of the simulation, it is necessary for learners to contact key individuals or organizations, in 
order to obtain additional information or receive clarification or insights into information they have already 
received. In the example shown in Figure 4, the learners are directed to contact fictitious company. The phone 
number for this fictitious company can be found provided on a web site provided by a link in a previous email. 

When the learners dial the correct number, they are presented with an audio phone conversation providing more 
information relevant to the simulation. 
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Dial the number for M5 Technologies 
to obtain additional information. 

The number for M5 Technologies can 
be found on their website. 

It is not necessary to dial a '1' before 
placing a long distance call. 



Please enter a phone 
number and press send. 
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Figure 4. Making Phone Calls 
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Another strategy used to simulate the real world environment is voice mails. Voice mails consist of text and 
narration describing the current contextual setting. The text typically describes which member the project team have 
an awaiting voice mail and directs the learner’s to click the star (*) icon on the keypad to retrieve the message (see 
Figure 5). When the learner clicks on the star (*) icon, the audio message was played. 
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BSD Communications Requirements Voice Mail 



The BSD Avionics IPT Lead has a voice 
mail. Click on * to listen. 
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send ^ 



reset £ 
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Figure 5. Voice Mail 



Phone conferences are used to present a dialogue between two people, or a group of people. The initial text 
introduces the context of the dialogue. The dialogue is then presented using both audio and text (see Figure 6). 




Figure 6. Phone Conference 
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Based on the information provided by the virtual interactions, teams perform specific analysis tasks and 
document the analysis on the PowerPoint templates provided on the simulation CD. Learners are directed to open a 
PowerPoint template on their team computer, then complete and document an activity or series of activities. In the 
example shown in Figure 7, the learner’s are directed to conduct a simplified trade study for thee engine options 
they have researched in the simulation. 
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SELP Design Simulation Trade Study 

Conduct a Trade Study 



You now have enough information to conduct a simplified Trade 
Study. Recall back in the pre course web-based module when 
you walked through a Decision Analysis Resolution (DAR) and 
Analytical Hierarchy Process (AHP). In this activity, you will 
apply the same principles. 

Open your PowerPoint file on your Team CD named 
5ELP. sim_Design_Team.ppt. Identify and analyze at least one 
risk for each Engine Option Risk slide in the file, and then 
complete the Trade Study slide for the three engine options 
specified. 

If your team has identified another engine option, feel free to 
add that option to your trade study. 

When you have completed this activity, click the Next button to 
continue the simulation. 
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Simplified Trade Study 
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Figure 7. Team Task 



After completing the activities, teams re-engage with the computer-based simulation and are presented with 
decision points. Each decision point allows the teams to select from multiple paths as shown in Figure 8. A 
significant feature of the decision points is that each potential decision has benefits and consequences, as shown in 
Figure 9, and each potential decision could be considered correct. Because of the number of variables and 
complexities involved with real world systems engineering problems, it is not practical to have the learners calculate 
actual numbers such as costs and schedule. To solve this limitation, potential benefits and consequences of the 
decision were presented on parametric graphs as seen on Figure 9. Regardless, the importance of each decision is 
not necessarily the potential benefits and consequences, but instead the process used to arrive at the decision. 
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Engine Option 



Engine Option 



Based on your evaluation of risks and the results of your Trade Study, which 
of the following options would you choose for moving from design to 
production? 



|T] Prototype - GE303 - IM507X 
0 Prototype -M 503 -M 50 7X 
|~C~| Prototype -M 50 7X 



Make your selection and click Submit. 



Figure 8. Decision Point 
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SELP Design Simulation Trade Study 

Decision Point 



E UR DECISION: Prototype - IVI507X 
^^econside^^^^^ontinu^ 



CONSIDER: 

This is the lowest cost option, but it will result in a significant production 
delay. Additionally, the M507X is the highest risk engine in the mix and if 
this engine is either significantly delayed or does not meet expectations, 
this option could seriously jeopardize the whole project. 



On the other hand, this option will greatly simplify the BSD fleet 
maintenance during the support phase. 




Figure 9. Benefits and Consequences of Decisions 



To help the learners validate their decision-making processes, teams debrief the entire class at the end of 
each simulation phase. Debriefing moves the learners from passive recipients of information to active participants in 
a shared experience (concrete experience). Debriefing also causes the learners to reflect upon and discuss the shared 
experience (reflective observation), consider how their experience and those of other learners relate to previously 
taught information (abstract conceptualization), and formulate methods for using the newly acquired knowledge in 
the workplace (active experimentation) (Walker, 2005). 
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Conclusion 



It is important to remember that the simulation is not the solution, but instead just a part of the solution. In 
order to meet the program goals, it was understood that the SELP program woidd need to facilitate learning along 
the entire hierarchy of Bloom’s taxonomy. The simulation, based on Kolb’s experiential learning circle, provides the 
learners with an opportunity to apply new knowledge and analyze the benefits and consequences of his or her 
actions. 

One of the first participants to complete the SELP has found a practical application for his training. A new 
contract requirement dictates that risk management costs be identified separately rather than being combined with 
other costs. The participant was able to apply the training he received in SELP to meet this need. He stated: "By 
applying a good risk management tool and the risk management techniques taught in the class, we now have the 
capability to provide the information to support the revised contracting process ... We took a process from 
Engineering and reapplied it to satisfy a contract requirement." 

In effect, the simulation provides a bridge (as shown in Figure 10) through the center of the Bloom’s 
hierarchy which prepares the learners for synthesizing the learning experience when conducting their on-the-job 
Systems Engineering project between segments three and five, and evaluating the impact of the learning experience 
during the post-course discussion and assessment. 




(Bloom, 1994) 

Kolb’s Experiential Learning Circle 
(Kolb, 1984) 

Figure 10, Tying the Theories Together to Bridge the Gap 
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Introduction 

Recent technological advances have redefined the traditional concepts in many fields and restructured the 
interactions in our social environments. The ways we communicate, learn and socialize are now different than once 
they used to be. The criteria for being “competent” or “successful” in our society are now measured with the extent 
we adopt new technologies into our lives, use them effectively and create solutions to the current problems. For the 
purpose of preparing competent and adaptable individuals to the complex and information rich society, educators 
need to reconsider their current educational practices by integrating emerging technologies into their classroom 
settings. 

High broadband connections and user-friendly development tools generated a new genre of internet users 
who actively contribute to the development of media by editing and uploading videos to the internet and sharing 
them with their audiences within a community. Videblogging has emerged as a way of communication through 
audio-visuals in blogging activities and gained extensive popularity among today’s Internet users by providing new 
ways of communication and representation of ideas through interactive channels. Although its popularity amongst 
Internet users has increased in the last couple of years, not a lot of educators focused on the use of videoblogging 
technology in education. As a highly developing trend on the Internet, videoblogging may offer teachers and 
students rich communicational channels for learning and communication of ideas through producing and sharing 
media. 

This paper aims at exploring the educational uses of videoblogging activities in teaching and learning 
settings. The intent is to introduce this highly interactive technology to educators by mainly investigating its possible 
uses for educational purposes. 



Blogging technology 

Weblog, “an internet based personal publishing system” (Miles, 2005, p. 1), has gained tremendous 
popularity among the Internet users in the last couple of years. Weblogs are frequently modified webpages where 
data entries are listed in reverse chronological order (Herring et al., 2004, p. 1). 

Armstrong et al. (2004) listed some of the characteristics of webblogging as follows: 

• Ease of use, where the author can publish to the web without using any programming code. 

• No need for installing any server software on the users' machine. 

• The user has extensive control over how his or her blog looks and operates. 

• Whenever the user edits his or her blog the results are instantly updated and available to others. 

• Like any other website, blogs can be simply linked to and navigated. 

Today, the majority of bloggers use webblogging as a form of personal communication and expression. 
According to Nardi et al. (2004) people blog for several reasons. Bloggers who participated in the study indicate that 
blogs serve as: 

• documentaries of their life through which they publish updated information about activities and 

events, 

• commentaries as a way of which people express their personal opinions and make their voices 
heard, 

• catharsis as a means of expressing feelings and emotions, 
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• muses by which people structure their thinking through writing, shaping their writing with their 
audience and creating an archive, 

• community forums in which bloggers share their pieces with others and reflect on other posts as 
well. 

“Combined with the unprecedendent opportunity, blogs provide for ordinary people to self-express 
publicly” (Herring et ah, 2004, p. 11). As blogs continue to grow in popularity, their unique characteristics have 
also been explored by educators to meet the learning needs of digital native young generation. Below, some of the 
possible benefits of using webblogging in educational settings are summarized from the literature. 

• Blogs can be used as electronic journals where students document their learning (Armstrong et ah, 2006). 
Writing reflections on field practices, students can record and report what they learned. 

• Blogs can be used to foster collaborative work. Bringing multiple users together to contribute and share 
their ideas in a platform, webblogs encourage students to have different perspectives through ongoing and 
interactive discussions and reflection on different ideas (Barlett-Bragg, 2003; Walker, 2005). 

• Blogs can serve as electronic publishing tools. Publishing their works, reports and artifacts in their blogs, 
students can make their work public, receive immediate feedback from their peers, practice and develop 
their writing and readings skills (Barlett-Bragg, 2003; Ward, 2004). 

• Blogs can support individual reflection. Webblogs provide students with opportunities to organize and map 
their learning progress (Xie & Sharma, 2004; Stiller & Philleo, 2003; Sharma & Fiedler, 2004). 

• Blogs can assist language learning. Communicative language learning can be enhanced with writings and 
readings in blogs (Ward, 2004). 

• Blogs can serve as student portfolios. Online portfolios have been used to enable students to organize their 
works and present them in a meaningful manner. Since webblogs archive students’ works automatically, 
they can help students see their processes through the postings at different times (Lohnes, 2004). 

What is videoblogging? 

Videoblogging is now a new and popular trend in blogging circle and differs from its parent, textual 
blogging, by offering audio-visual capabilities to communicate. EDUCAUSE Learning Initiative (2005) defines a 
videoblog as a “web log (blog) that uses video rather than text or audio as its primary media source” (p. 1). 
Borrowing from the definitions of Bryant (2006), this paper will use the term videoblog to refer to blogging activity; 
vlog to the posts and videos; and videobloggers to the people who produce and publish vlogs in their blogs. 

Advances in broadband connections and video-hosting services, emergence of easy-to-use video editing 
software, affordable prices of video recording tools and accessible video technologies stimulated videblogging 
diffusion on the Internet. As Hoem (2004) affirms, “the importance of video, being a very powerful medium, the 
increased amount of video material on the web, and the possibilities offered by weblogs when it comes to 
collaboration sums up in ‘videoblogs’ as one of the most promising tools which may foster media literacy” (p. 3). 

Vlogs combine images, audio, movies and text to communicate personal reflections, expression of feelings, 
documentaries and stories. Videobloggers generally edit their video with video editing tools, adding background 
music, sound or text and upload them to their videoblogs to share their vlogs with their audiences. Videobloggers 
share their videos within a community, receive comments on their videos and set a conversation with their 
audiences. 

The videoblogging process is relatively simple for anyone who is familiar with video editing tools and 
blogging. A videoblogger first makes a short film, transfers the film from digital camera to computer, edits it with 
video editing software, becomes a member of a website which hosts video, uploads the video to the website, gives a 
title to the videoblog, shares it with the audience in the videoblog and gets feedback on vlog through comments. The 
quality of vlogs ranges from very basically edited vlogs to the professionally edited and esthetically created ones 
based on the technical skills of videobloggers and the intended message of vlog. 

Reasons for videoblogging 

People create vlogs for several reasons. “Videobloggers are artists, filmmakers, technology geeks and 
citizen journalists who go out and report news that major news networks may have overlooked or underreported” 
(Bryant, 2006, p. 12). The types of videblogs range from videoblogs on personal lifes to videoblogs about news, 
politics, environmental problems, magazines and entertainment. The power of video in delivering the message 
provides people from variety of backgrounds and interests to share what they feel and think in regard to a variety of 
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subjects. Bryant (2006) lists some of the reasons for videoblogging: Keeping in touch with distant friends and 
relatives; ego-stroking; archiving family history and interviews; ranting about issues in their lives; learning more 
about digital video technology; increasing the online presence of a minority group; making people laugh (for 
comedy vloggers); making new friends online; to make money (videblogging as employment) and to promote 
business as a means of delivering products (pp. 16-17). 

In his videoblog travelvlog.blogspot.com . Graham Walker shares his vlogs about his trips around the 
world. Steve Garfield as a pioneer and supporter of videoblogging, creates vlogs about anything at 
stevegarfield.blogs.com/videoblog/ and produces “'The Carol and Steve Show” which displays short episodes from 
his daily life with his wife Carol. "It's part of the democratization of media," says Mr. Garfield. "It gives people the 
tools to express themselves and be creative in all different ways" (Searcey, 2005). 

Videoblogging may serve as a platform for ordinary citizens to be journalists and reflect on what they see 
as essential for others to know, listen and see. Josh Wolf posts in his videoblog “the revolution will be televised” at 
http://thisrevolution.blogspot.com/ at least once a week to tell the stories that they believe are ignored by 
conventional news producers. The aim of the videobloggers like Wolf is to democratize media by promoting citizen 
journalism, the idea of promoting ordinary people to “report and produce news and entertainment, using new 
technology that gives them powers once held solely by the Fourth Estate” (Hua, 2005). Another videoblogger 
Raymond Kristiansen states that “We don't need the mainstream media to tell us what's interesting anymore. 

Millions of people want to tell their stories, and with videoblogging, they can tell their stories” (Boyd, 2005). 

The number of videobloggers has increased in the last couple of years due to the abovementioned reasons 
and they are inspired by their audiences who are watching their vlogs and making comments on them. Communities 
are formed among videobloggers who advocate videoblogging such as “meet the vloggers: a series of focused on 
videoblogging” at http://meetthevloggers.blogspot.com . Vloggercon at http://www.vloggercon.com is another event 
that videobloggers come together to share, interact and collaborate with other videobloggers. 

Videoblogging is considered as the next generation of communication through posting, sharing, producing 
and publishing content on the Internet. Among videobloggers are some educators who have been trying to explore 
the educational benefits of videblogging. This new and diffusing technology is being experimented to enhance 
educational settings by promoting active participation of learners in their learning process. The next section aims at 
informing educators about the educational benefits and possible educational uses of this fast growing technology 
trend and phenomenon. 

Videoblogging in education 

Videoblogging is one of these latest technologies that opened new ways of communication through public- 
created media. Since videoblogging is such a new technology and a trend on the Internet today, only few 
videobloggers have recognized the promises of videoblogging for teaching settings and have used it in their teaching 
practices. The teachers and educational practitioners who are interested in this technology come together in a forum 
“Vlogging for K-12 Teachers” at http://groups.google.com/group/Vlogging-for-K-12-Teachers , discuss their 
practices, share their works and find out other possible educational uses. Two interviews were conducted with these 
practitioners who have been using videoblogging technology in their educational practices since the very early days 
of videoblogging technology. The insights, ideas and experiences gathered in the interviews are summarized in two 
sections: Teachers as videobloggers and students as videobloggers. 

Teachers as videobloggers 

In videoblogging community, there are some teachers and educators as videobloggers who want to 
communicate their educational messages with this highly engaging tool. Among them was Bre Pettis who now 
makes magazines doing videoblogging. He generates videoblogs about the processes of making things in a 
humorous way. He put his video on the Internet in September 2003 and started videoblogging in the Spring 2004. 
He was keeping a textual blogging before and was later attracted by the offerings of new videoblogging technology. 
Since then, he has been keeping a videoblog about the things he creates and teaches his audience how to make those 
things at http://imakethings.com . He explains his reasons for videblogging: 

I am an artist and I wanted to share the “I love you project” with more people than could see it in a 

gallery. . . I have two jobs in life, being creative and supporting other's creativity. 

Last year he was a middle school teacher and used videoblogging in his art and writing courses. Mr. Pettis 
formed http://room32.com to teach people and to give his audience more opportunities for self-expression. He 
created those vlogs to share them with his students, parents and other colleagues. He created many vlogs to inform 
students about their future projects in the class, collecting their works in a video and share them with the 
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community. He prepared a videoblog “in the future” in which his students made some predictions about the future. 
Mr. Pettis comments on his experience: 

That was just something to do in the last 10 minutes of class one day. You can tell they felt comfortable 
taking creative risks in my class, everything from living on the moon to we will all die, world war three. . .1 
am proud of that video, it was really the best collaboration and a lot of my videos for room 132 are news- 
style updates with student work. 

Room 132 



MO ' 4 10 vtn-o" Wh«t doo the of to*no»f ow pf«d<t the future be 

like? Weteh rfttt video to find out I'm re«i*r proud of t»«» video end the tok and jM,tB 

my v*v lottery ytudroiv' More (Otnmmtl over on Mr Pcttn' pet tonal blag. I Mat r .. 

Figure 1 - "In the Future... ” vlog at http://rooml32.com/2006/03/12/in-the-future/ 

Mr. Pettis recorded those videos with the help of his teaching assistants. He explains his reasons for doing 
vlogs in the classroom: 

For me as a teacher the important parts were to document what I was doing and share it, let the parents 
know what was going on in class, and getting a bigger audience than my 150 students. 

Mr. Pettis shared students’ works on Animated Flip Books by doing vlogs about them and publishing at 
Room 132. Parents, other teachers and students watch this vlog and make comments on it. 



Room 132 Animation Flip Books 



November 1 9, 2005 at 1 0:50 am Filed under Room 1 32 News, video 



4) ii ~ * ”<i ik 



Select Player 





Figure 2- Animation Flip Books Vlog at http://rooml32.eom/2005/l 1/1 9/room- 13 2-animation-flip-books/ 

According to Mr. Pettis teacher-created videoblogs and student-created videoblogs are two different 
educational practices and they have different dynamics in their applications. He stresses the impracticality of 
student-created videoblogs in schools which have limited technology infrastructures. 

Students as videobloggers 

Jonny Goldstein is one of the leading practitioners in videoblogging field trying to promote the use of 
videoblogging in educational practices. He is also an early adopter of videoblogging technology, publishing his first 
videoblog in December 2004, the time when people started doing videoblogging. He started doing videoblogging for 
personal reasons after he attended vloggercon (a yearly conference aimed at bringing videobloggers together, 
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http://www.vloggercon.com ') held in December of 2004 and he “fell in love” with this new technology. He talks 
about his reasons for videoblogging: 

It was a natural for me. I already was text and photo blogging partly because I enjoy expressing myself in 
text and images, partly to let my friends and relatives stay up to date on what I was doing, and partly to 
professionally promote myself starting in autumn of 2004. 1 was starting to perform a lot of stand-up 
comedy so it made sense for me to make funny videos to get my stuff out there. 

He started doing videoblogging for educational purposes on October 2005. Although he was not a certified 
regular classroom teacher he had been working intermittently in various education-related jobs, working as a 
substitute teacher in the New York City public schools and as a theater, art, and music teacher in an afterschool 
program for a couple of years. His videoblogging practices for educational purposes started with his job as a project 
director for a private company, which was in the business of doing robotics, interactive multimedia, digital audio 
and digital video with children. The BX21 Videoblogging Project is made by Vision Education and Media and 
SOBRO and is financed by a 21st Century Community Learning Centers Grant ( http://vemnyvlogs.org/bx21/ ). The 
aim of the project was to make students to get ready for the work world and to connect school and real life with the 
technology component. Mr. Goldstein directed this project and worked within the campuses of five of the schools 
involved in the project. 

Students were volunteers and it was free and most of them were low income. Class size was varied from 15 
to 4-5 active participant kids who would meet two afternoons a week. Kids usually did the videos as group 
projects so what was the aim of the program. 

According to Mr. Goldstein the educational benefits of videoblogging are developing students 
communicational skills through storytelling, enhancing their visual literacy, improving their technical skills and 
learning how to blog to use a social software. The topics of the vlogs that students prepared as part of this 
afterschool project reflected the very benefits of videoblogging on student self expression and learning. For instance, 
a student prepared a short clip for a documentary which was about attitudes toward homosexuals by conducting 
interviews with his classmates, the other student prepared a video clip to do a review of a game and others did an 
opinion piece about school lunch. 

Mr. Goldstein also talks about some of the challenges they encountered during the videoblogging activities 
in classrooms such as technical difficulties with uploading videos in the school labs due to the security settings , 
getting releases from the parents to put vblogs online and paying attention to ethical issues such as safety and 
privacy. Mr. Goldstein summarizes the precautions to be taken while incorporating videoblogging activities into 
classroom practices effectively: 

• Have a good release prepared and get it signed. 

• Discuss what is and is not appropriate with the students before they put work online. 

• Discuss the Internet safety with students. Ask students not to use their last names and let you know if 
anyone left them an inappropriate comment or tried to contact them on the Internet. 

• Review students’ work before they post it online. 

Although using videoblogging in classroom settings poses some challenges to teachers, today the young 
generation is much more comfortable with the technology, and technology is already integrated into their lives. In 
order to meet students’ needs, educators need to be familiar with and incorporate new technological advances into 
their classroom practices. Mr. Goldstein makes some predictions about the future of videobloging: 

Teens or college students now are going to be way more comfortable with technology, “Do-it-yourself 
media social software”, so it's going to shift; we're just at the beginning. More wireless connectivity, 
cheaper and better cameras, higher bandwidth connectivity. . .This means more people will be shooting 
video and putting it online everywhere. . .1 think schools have got a big challenge, they are already in, 
because technology is changing so fast and their students are going to be the experts 

Mr. Goldstein considers teachers’ role in videoblogging activities as being guides who facilitate students’ 
videoblogging activities. Creating a learning community in which both students and teachers learn from each other, 
communicate and contribute to the teaching-learning process, videoblogging is likely to engender valuable 
educational outcomes. Mr. Goldstein at “We are theMedia” blog 

( http://wearethemedia.com/2QQ5/10/25/videoblogging-in-education ) gives some examples of educational use of 
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videoblogs. He also proposes that teachers might place videoblogs on the Internet in a way that only selected 
viewers can have access to them with password-protected blogs. Therefore, teachers will be able to reduce security 
problems and feel much more comfortable in incorporating videoblogging into their classroom activities. 



Videoblogging as a promising educational tool 

Videoblogging offers rich experiences both to the teachers and students who want to communicate with movies, 
sounds and images, reflect on what they teach, learn and think and share them with a community. The possible 
educational benefits and uses of videoblogging activities are summarized below: 

• Meeting the learning needs of students 

Today’s digital native students are born in this technology-rich society and have had closer contact with 
multimedia tools already. What they see and perform outside of the schools should be reflected in educational 
practices to answer their learning needs. Videoblogging can be used as a tool for meeting the needs of students from 
diverse backgrounds. Learners come to learning environments with their prior knowledge, unique characteristics and 
beliefs. Incorporating videoblogging activities into classroom settings will create opportunities for children to build 
their knowledge onto their unique prior understandings and different learning styles. 

• Videoblogs as a tool for reflection on learning: Students’ cognitive abilities can be enhanced with 
videoblog reflections, in which they come to realize what they know, what they learned and what they can 
learn in the future. 

• Videoblogs as a way of demonstrating understandings about variety’ of concepts: Videoblogging can be 
used as a way to express variety of subjects and concepts. Learning by design approach can be used with 
videoblogging activities in which students create the meanings through the design processes of vlogs. 
Videoblogs can be part of design projects in which multiple intelligences are used effectively, students’ 
higher order thinking skills are developed, and problem solving abilities are enhanced through the creation 
of a product for a real audience (Kahn & Ullah, 1996). 

• Videoblogs as a tool for collaborative work, creating learning communities: Hoem (2004) stresses the 
importance of the function of videoblogs as wikis on the creation and production of collective videos. 
During vlog production phase, students may work together to design their documentaries and enhance that 
process by sharing their work on videoblogs with other students, even the people outside of the class. Their 
learning space can expand the borders of a classroom, creating new types interactions between them and 
the society. 

• Videoblogging for digital storytelling: Digital storytelling has been considered as a powerful method for 
engaging students and teachers through telling their stories with multimedia tools such as images, audio, 
video and web technologies. “The use of videoblogs for digital storytelling may be one way to encourage 
strong student participation in e-portfolio projects” (EDUCAUSE, 2005). 

• Videoblogging for improving media literacy: Today we are surrounded by messages that are being 
conveyed in variety of forms. Media tools have been evolved over time from printed materials to highly 
interactive channels such as audio-visuals and the Internet. To become competent in this highly rich 
communicational environment, to understand and interpret messages and to create the messages by using 
media tools, students should have media literacy skills. Videoblogging may serve as a strong educational 
tool by providing students with rich opportunities to reflect on messages being conveyed and in the creation 
of their own messages. 

• Videoblogging and student journalism: Citizen journalism has emerged with new technological advances 
that allow ordinary people to be actively involved in the creation and dissemination of the information and 
news to the rest of the world. Instead of passively demanding ready-made news, now anyone with some 
basic internet knowledge can be a journalist who can share his/her personal information or news with the 
audience. Encouraging students to be citizen journalists will help them understand their environment better, 
criticize ready-made structures and thus take the very first steps of being active citizens in the democratic 
society. Videoblogging when used as a tool for student journalism can be a powerful tool for helping them 
convey information and news that they think important for others to watch, think and reflect on. 
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Videoblogging and Technological Pedagogical Content Knowledge (TPCK) 

In order to overcome the challenges that teacher face today in integrating integrate technology in their 
classrooms appropriately. Technological Pedagogical Content Knowledge (TPCK) has emerged as a new vision in 
teacher education field addressing the “form of knowledge that goes beyond all three components (content, 
pedagogy and technology)” (Mishra & Koehler, 2006, p. 1028). TPCK framework focuses on technology integration 
into specific curricular topics with the selection of particular pedagogical techniques. Teachers need to consider the 
interactions between technology, pedagogy and content in their videoblogging classroom activities. Bull et al. 

(2007) state that “digital video technologies offer unique opportunities (through interactivity and user-generated 
content) to rethink the instructional paradigm particularly to match the needs of the subject to be thought” (p. 3). 
They give specific examples of the integration of digital video technologies into different disciplines with different 
pedagogical goals. 

Table 1 

An Illustration of Linking a Technology (Digital Video) with Pedagogical Goals in Different Content Areas (Bull et 
al., 2007, p. 6). 



Discipline 


Content 


Technology 


Pedagogical Goals 


Science 


Physics 


Digital video 


Rectifying Naive 
Conceptions 


Physics 


History 


Digital video 


Supporting Historical 
Inquiry 


Language Arts 


Reading 


Digital video 


Reinforcing Visual 
Imagery 


Mathematics 


Trigonometry 


Digital video 


Connecting 

Representations 



Similarly, videoblogging technology can be integrated in different content areas with different pedagogical 
goals with a focus on its affordances of publishing and sharing digital video content on the Internet and constraints 
of safety and privacy issues. If teachers consider the interactions between pedagogy, content and technology in their 
videoblogging classroom activities, they can offer unique learning opportunities to students. 

Conclusions 

Videoblogging activities can meet the needs of today’s students who are surrounded with these highly 
dynamic and interactive technologies already. Students’ interaction with producing and publishing media and 
sharing it with a community can improve their communicational and technical skills which are required for them to 
be successful and competent citizens in the society. When used effectively in educational settings, videoblogging 
may redefine classroom borders expanding learning experiences outside of the school to the community connecting 
school, students, teachers, administrators and parents in a learning community. 

Videoblogging as an emerging tool, a popular trend on the Internet and a new way of media production has 
a lot to offer for educational settings. Without any doubt, more and more educators will start to experiment this 
engaging tool in the teaching and learning settings in the near future. 
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The public schools in New Orleans have long been believed to be among the worst in any big city school 
system in the United States. The students are the poorest citizens of one of the poorest cities in one of the poorest 
states in the country. While the population of the city is approximately 65% African American, the student 
population in the district is over 90% African-American- a result of continuously failing schools and decades of 
white flight. In a city with one of the highest rates of private school attendance in the country, 25% compared to 
10% nationally (Newmark & De Rugy, 2006), the public schools have been consistently under-funded and 
neglected. Prior to the storm, the district was led by ten superintendents in ten years and several have left under 
clouds of suspicion regarding mishandled money. Until the eyes of the nation fell on this city after Hurricane 
Katrina struck on August 29 th , 2005, there had been little political will to improve the failing system. 

The nation cringed at the TV news clips of poor African Americans huddled, dehydrated, and dying at the 
convention center and Superdome waiting for handouts from the government. The embarrassment to the city was 
real and this may provide the political will to substantially change the public schools in this city. In the aftermath of 
Hurricane Katrina, the district temporarily lost 100% of its students and didn’t open a single public school for more 
than two months. As it became apparent that the school board was not prepared to bring the district back from such a 
devastating blow, reformers began to see a silver lining in Katrina’s dark clouds. State School Board member Leslie 
Jacobs said, “The diaspora of New Orleans represents the opportunity to rebuild our public school system” (Inskeep, 
2005b). While some saw this as an opportunity to rebuild the system, huge segments of the population were living in 
Houston, Baton Rouge, Atlanta and hundreds of other places. With the low-income, non-white residents often most 
affected by the storm, it fell to the educational organizations in the city to lead the debate over how the schools 
should be reopened. Could these organizations follow through on their promise to improve schooling for the lower- 
income residents of New Orleans? With leadership structures and educational platforms already in place, these 
stakeholder organizations (teachers union, school board, etc.) were well-situated to influence the course of events in 
the tumultuous few months following the storm- would they seize the opportunity? This paper addresses the 
question: How did stakeholder organizations respond to the post-Katrina collapse of the New Orleans Public 
Schools? Components of this question are: 

• Why did groups act when they did? 

• What basis did they use to argue for their proposed reforms? 

• What successes/failures have groups had in bringing their educational vision to 
fruition? 

In understanding the ways in which various stakeholder organizations responded to Katrina, it should be 
possible to understand the reasons for the success and failure of certain groups. This may give some insight into the 
complex social and political forces that surround any effort at changing urban schools. While Seymour Sarason has 
noted “the intractability of schools to educational reform” (Sarason 1990, p. 147), it is possible that schools are not 
intractable as much as reformers are ignorant of the complexities of changing organizations as complicated as urban 
schools. Understanding the role of stakeholder organizations in post-Katrina reforms is one small step towards the 
larger agenda of understanding how urban schools change. 

Theoretical Framework 

Michael Fullan (2000) addresses the concepts of cultural and structural change as they apply to school 
reform. Structural changes involve changes in governance structures (Cuban & Usdan, 2002) and formal leadership 
roles in an effort to improve the educational outcomes of a district. Cultural change, in this case, involves the active 
participation of many individuals in the creation of classrooms into places where learning is a shared goal and 
shared responsibility. Ideally, this type of change involves schools that utilize professional learning communities to 
examine student work and make appropriate alterations to teaching practice (Newmann and Welhage, 1995; 
Schlecty, 1990). While cultural change is generally a more powerful agent in the reform of systems of education, it 
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is also the most difficult. Cultural change without a supportive governance structure is both challenging and 
exhausting (Vibert & Portelli, 2000). Stakeholder groups, especially in the case of post-Katrina New Orleans, are 
able to influence district structure and make it either accommodating or hostile to the development of professional 
learning communities which lead to deep cultural changes. This study will examine the roles of five stakeholder 
organizations and their success in implementing structural reforms in the post-Katrina redesign of the New Orleans 
Public Schools. 

Five Key Stakeholder Organizations 

The roles of five key stakeholder organizations will be explored to answer the research questions posed 
above. While this is by no means an exhaustive list of such groups, they represent some of the more influential 
groups in terms of their ability to influence the massive post-Katrina restructuring in the district. A brief description 
and abbreviation for each organization is given below. 

1. United Teachers ofNew Orleans (UTNO) 

UTNO is the AFT affiliated teachers union that counted approximately 4700 dues paying members pre- 
Katrina and less than 300 after the storm. 

2. Orleans Parish School Board (OPSB) 

The seven-member board that was directly responsible for the operation of 1 1 7 schools prior to Katrina and 
currently oversees the operation of 5 schools. Members are elected to 4 year terms by residents of 7 geographic 
districts in the city. 

3. State of Louisiana 

This organization consists primarily of state legislature, the governor, the State Superintendent, and the 
Board of Elementary and Secondary Education (BESE) which contains 8 elected members and 3 members appointed 
by the governor. The BESE Board is directly responsible for 102 (mostly closed) New Orleans campuses under the 
control of the state-run Recovery School District (RSD). 

4. Algiers Charter School Association (ACSA) 

A community group that created a collection of 14 charter schools shortly after Katrina. This group 
enrolled about one in five New Orleans Public School students as of September 2006 

5. Mayor Nagin’s “Bring New Orleans Back” Education Sub-committee (BNOB-ED) 

Committee of nearly 70 education policy elites formed by the mayor to develop and present a proposal for 
the rebuilding of the New Orleans Public Schools. 

Data Collection and Analysis 

As a former teacher in the New Orleans Public Schools, I began collecting media reports of the post- 
Katrina school reforms immediately after the storm. As it became clear that the school board would not be able to 
retain control of the district, and that significant reforms were imminent, my collection of media reports became 
more systematic. Approximately 150 media reports (both print and radio) have been collected to trace the activity of 
stakeholder organizations post Katrina. Some federal and state government documents and recordings of public 
meetings focusing on the school system supplement these media reports. Additionally, public information released 
by these organizations via press releases, publications, and Internet sites has been reviewed as important primary 
sources. 

Analysis of this data consisted of chronological analysis of the major actions and public statements of each 
stakeholder organization which indicated the general position of each with regard to changes within the district. 
When the general position of each of the five groups included here are taken as a whole, it is possible to see which 
groups have been more effective in translating their vision into reality. Of course, events are still unfolding rapidly 
in New Orleans, and the groups that have had success influencing reforms at press time are not necessarily the 
groups that will have the most long-term success. The untimely death of state Superintendent Cecil Picard and the 
hiring of Philadelphia superintendent Paul Valias to run the RSD are some of the latest developments. A longer- 
term study of the results of post-Katrina reforms would certainly help in seeing the true extent of the changes that 
Katrina brings to the New Orleans Public Schools. 

Responding to Disaster 

This section traces the actions of five stakeholder organizations in response to Hurricane Katrina on August 
29, 2005. In addition to the important actions of each group, a summary of its plan for post-Katrina schooling will 
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conclude each section. In some cases, this plan is a written document which leaves little need for interpretation. In 
others, the plan has been pieced together and interpreted from organizational actions and public statements. 

Stakeholder #1- United Teachers of New Orleans (UTNO) 

UTNO called for a quick resumption of classes following the storm. They linked the reopening of schools 
with the return of population that would galvanize the citywide recovery process. Initially disregarding calls from 
other groups that extensive redesign of the schools should precede the reopening of the city’s overwhelmingly 
failing schools, union president Brenda Mitchell preferred “that all attention is on the resumption of instruction. . .” 
noting that radical changes to the school system “should be put off until a later date” (United Teachers of New 
Orleans, 2005a). Mitchell admonished the school board’s rapid creation of charter schools, blaming the discussion 
around charters for the delay in reopening public schools for the city’s children. The union filed suit in city court on 
November 8th to force the district to reopen a small number of schools (Ritea & Warner, 2005) 

At the state level, Louisiana Federation of Teachers President, Steve Monaghan vehemently opposed a 
proposal that would have created a state-funded voucher program (United Teachers of New Orleans, 2005b). In 
November 2005, while the OPSB was busy chartering schools and the state legislature was beginning to consider a 
takeover bid in the upcoming special session, UTNO released its vision for the re-opening of schools in the flood- 
ravaged city. Their plan called for the rapid opening of schools that would maintain the same organizational 
structures and curricula that were in place prior to the storm. The changes they focused on were related to working 
conditions (class size, building cleanliness) and a call for more ethical behavior from district officials (United 
Teachers of New Orleans, 2005c). Arguing that the majority of the public school community was physically absent 
from the city, the union consistently argued that any large departures from the pre-Katrina system would be unwise 
due to the unavailability of community input. 

While this plan serves as an ideological snapshot of what the union wanted the rebuilding process to look 
like, it quickly became a meaningless plan. UTNO lost nearly all of its power when Governor Blanco signed House 
Bill 121 that swept over 90% of New Orleans Public Schools into state control- voiding the bargaining agreement 
that the union had in place with the school board. The union was left to watch passively as the district fired nearly 
95% of its dues-paying members in February’s post-Katrina budget crunch (Ritea, 2006b). In fact, when the 
American School Board Journal did a feature story on the recovery process of NOPS, they included articles on six 
different stakeholders in the new district, but union leadership was not one of them (Dillon & Vail, 2006). In 
September 2006, the school board voted to cease payments to UTNO’s Health and Welfare Fund that defrayed 
medical costs incurred by UTNO members (Ritea, 2006e). 

In the end, UTNO sought a reopen-first, reform-second strategy that ignored the desire of the public to 
overhaul the NOPS in the wake of Katrina. Their calls for the “renaissance, not replacement” of the system sounded 
to many citizens like calls for the status quo and left it with few allies and little political power following the state 
takeover. Time will tell if workplace issues in the charter-dominated district will bring about a resurgence of the 
teacher’s union. 

Stakeholder #2- Orleans Parish School Board (OPSB) 

The OPSB was in dire straits even prior to the destruction caused by Hurricane Katrina. It had hired 10 
superintendents in 10 years and diverted money from the classroom to hire a New York consulting firm Alvarez and 
Marsal to help track down $71 million in unaccounted for federal funds (Inskeep, 2005a). Public trust in the school 
board was low, and recent scandals involving missing money, continued school violence, and poor academic 
performance had many people plotting for a way to remove the district from the board’s mismanagement even 
before Katrina. 

After the storm, the school board first met on September 15 th in Baton Rouge. At the meeting. Board member 
Phyllis Landrieu suggests inviting former Secretary of Education Rod Paige and retaining Bill Roberti from Alvarez 
and Marsal to take over leadership of the district (Orleans Parish School Board, 2005). The vote failed, but this 
shows the school board’s lack of confidence in its own abilities to successfully reopen the district. During the same 
meeting they placed all employees on disaster leave and set a target date of November 1 st for the first school 
reopening. In a forshadowing of things to come, the board also approved a charter for the Lusher Elementary School 
in a 5-1 vote, The move to decentralize the district was begun by the very people who would have been expected to 
attempt to maintain control. 

At their next meeting on October 28th, the board approved charters for an additional 20 schools with the 
condition that 20% of the students at each chartered school be eligible for free and reduced lunch (Ritea, 2005a). A 
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federal grant of some $2 1 million forced the hand of the cash-strapped board to accept the charter proposals despite 
the feeling expressed by board president Torin Sanders that this decentralization could lead to problems of equity in 
the district. The district’s chartering of its schools was also a response to harsh public criticism for not opening 
schools when private schools in the city and public schools in hard-hit St. Bernard Parish were already open. 
Superintendent Ora Watson apologized for the slow opening of schools and pushed the target date for opening back 
from November 1 st to November 14 th due to district financial problems. 

Eventually on November 28 th , the first public school in the city was opened, nearly two months after the 
storm struck (Ritea, 2005c). Three days later, just like UTNO, the school board was stripped of nearly all of its 
power when the state took over 102 of it’s 117 campuses. Of the remaining 15 schools, seven already held charters 
with the district, leaving the board in direct control of only 8 schools (Ritea, 2005d). The board, with the state’s 
help, had removed itself from any important role in the short-term future of the New Orleans Public Schools 

The district was left with only cleaning up to do once the state takeover occurred. The once-behemoth 
district now had 61 central office employees instead of 1,200 and operated 3 functioning schools instead of 1 17 
(Ritea, 2006a). In March, the district tried to recover some costs by selling some of its ruined school busses on Ebay 
as Katrina collectibles (Ritea, 2006c). In April 2006, superintendent Ora Watson announced that she would not seek 
an extension of her contract after it expired in July. This guaranteed that the much-smaller district would be led by 
its 1 1 th superintendent in a decade. 

Stakeholder #3- State of Louisiana 

Six weeks after Hurricane Katrina struck New Orleans, Jay Dardenne, a state senator from Baton Rouge 
said, “I don't want to rebuild those failing schools” (Anderson, 2005a). It was clear that state lawmakers saw this as 
a golden opportunity to rid the state of an under-performing school district that had been an embarrassment for 
decades. In the state’s most attractive city for outside investment, business balked at moving to New Orleans partly 
because of the decrepit condition of its public schools. Leslie Jacobs, an at-large member of the state BESE Board 
stated, “It's hard to find a silver lining from Katrina, but one silver lining is that the school board can start anew. 

And if any school district needs to start anew, it's Orleans" (“New Orleans wants to rebuild schools”, 2005). With 
public support for the school board waning as schools opened up in other hard-hit areas, the state passed House Bill 
121 on November 30 th , which moved 102 of New Orleans’ failing schools into a state-run Recovery School District 
(RSD). The state would have until May to submit a plan for operating the 102 schools. After five years, progress 
would be assessed and some schools could be returned to the board. At the bill signing ceremony. Governor Blanco 
summarizes the state’s position well: "We see an opportunity here to just do something that is incredible" 

(Anderson, 2005b). Depending on one’s point of view, that something incredible could be dismantling the 
professional community of the New Orleans Public Schools, or ridding the city of an inefficient bureaucracy and 
paving the way for change. In either case, the state was moving forward, and with its constitutional power, there was 
little meaningful opposition to the takeover. Change, in some form, was imminent. 

Stakeholder #4- Algiers Charter School Association (ACSA) 

A mere five weeks after the storm, ACSA petitioned the board to create charter schools in the relatively 
undamaged West Bank section of the city, which is located across the Mississippi River from downtown and did not 
get the standing water that caused most of the damage in other areas. The school board approved a single charter, 
granting control of all 13 public schools on the West Bank to the Algiers Charter School Association. The ACSA 
board, consisting of New Orleans educators and West Bank residents, approved a $22.6 million budget on 
November 4 th . This budget called for opening up eight schools by January for up to 5,400 students on a citywide 
open-enrollment basis (Ritea, 2005b). ACSA, like OPSB, hired New York turnaround consultants Alvarez and 
Marsal to help plan their recovery process. After some delays, schools opened on December 14 th with 1,324 students 
attending classes at five schools. OPSB had opened its first school on November 29, but by September 2006, ACSA 
would expand to eight schools and serve nearly one-in-five NOPS students. Despite critiques that the lack of a union 
contract and ACSA’s utilization of a basic skills test for teaching applicants (Polier, 2006), this district-within-a- 
district has had fewer of the problems with supplies, staffing, and student violence that have plagued the state-run 
schools. Their vision provided an appealing alternative to many, combining local control with promises made by 
long time New Orleans educator and ACSA CEO Brian Reidlinger that these schools would be different than they 
were pre-Katrina. 
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Stakeholder #5- Mayor’s Bring New Orleans Back- Education Committee (BNOB-ED) 

New Orleans Mayor Ray Nagin was a fixture in the national media following the Katrina disaster and 
gained notoriety for the abusive language he directed at the Bush administration and his “chocolate city” comments 
aimed at appeasing black residents who felt they were being pushed out of the rebuilding process. As a part of his 
rebuilding efforts, he created seven volunteer committees to guide the city’s initial actions following the storm. One 
of these committees was the BNOB-ED led by Tulane University president, Scott Cowen. This group had 20 
members on its steering committee, most of whom were business and education elites in the city. The biographies of 
this group, ranging from university administrators, to K12 principals, to state education officials led them to undergo 
a thoughtful process of examining what the storm meant for NOPS, researching other urban school reform efforts, 
and proposing a detailed report on suggestions for reforming the district in the wake of Katrina. 

They held three public meetings in New Orleans that encouraged public participation. Their first meeting on 
November 19, 2005 attracted about 50 participants (Bazile, 2005). In addition, they collected school improvement 
data from 1,500 parents, teachers, and students through telephone and electronic communication with displaced 
residents. At this meeting, committee chairperson Scott Cowen noted the long-term aspect of their mission: “Most of 
the dialog has been about who will control the schools, not who will plan for the school's future” (Bazile, 2005). The 
committee worked though November and December to draft a plan, eventually dubbed the “educational network 
model” that they believed would address some of the inefficiency and inequity of the old system. After several 
public meetings to collect feedback on the plan, it was submitted to the Mayor and presented for public comment on 
January 17 th , 2006. Of course, during the process of creating the plan, control of the district had moved away from 
the school board and towards the state and the charter operators. So instead of pitching their idea to a single 
institution, they had to address a patchwork of groups that all controlled pieces of a somewhat balkanized 
educational landscape. Scott Cowen’s worries about battles for control superseding educational improvement 
seemed quite appropriate at this point. While their educational network model included input from many residents 
and adopted successful practices from other urban districts, without formal power, their plan simply died. The state 
may, in the future, decide to implement portions of the BNOB-ED plan, but it also has complete authority to ignore 
it. 



Findings 

Based on this analysis, the reactions of stakeholder organizations to Hurricane Katrina appear in several 
cases to have somewhat more to do with maintaining or winning political control of the school system than with 
improving the academic success of New Orleans students. The self-interest of many of the stakeholder groups and 
their need for self-preservation or political influence is the central finding of this study. This finding is consistent 
with other studies of curricular change (Kleibard, 2004) and those who study complex social systems (Hutchins, 
1996). But this is not a story of mindless land grabbing by these stakeholder organizations. There may be a specific 
effect of Fullan’s (2001) structure/culture dualism at play here in that the organizations who were successful in 
gaining access to the structure of the district were, in most cases, those who proposed a new vision of the culture of 
the school system. Perhaps the conservative nature of the educational system maintains its current structure unless 
overwhelmingly convincing visions of a new organizational culture can be expressed by a particular group. An 
examination of each of my three original research questions shows this in more detail. 

Findings #1: Why did groups act when they did? 

It appears that stakeholder groups with traditional power (school board/union) did not act immediately. 
They were slow, bureaucratic organizations that were unprepared to lead the district prior to Katrina, and even less 
prepared to do so after. Contentious relationships between the school board and the union prior to the storm left the 
city without any unified political coalition that could organize around the rebuilding efforts of NOPS. Despite some 
posturing that both groups intended to lead the rebuilding of the school district, neither did anything significant and 
both are now significantly weaker organizations than they were before the storm. These two groups failed to act 
because they were bureaucratic monopolies and weren’t accustomed to doing anything at all- even in the best of 
circumstances. 

ACSA and mayor’s group formed when it became obvious that the board couldn’t re-open the schools. In 
the silence cause by the school board’s slow response, these two new groups formed- desiring to significantly 
reshape the system of public education in New Orleans. To be sure, ACSA was spurred on by the federal 
government’s announcement of a $21 million grant to support charter schools in hurricane affected districts- a 
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highly political move consistent with the Bush administration’s market-based school reform policies. But ACSA 
also formed in response to a desire to get out from under the stifling bureaucracy of the top-heavy school district. 
They were escaping the ineffectual policies, which their charter application refers to as 
“a one foot thick set of documents that have not been reviewed for consistency and necessity in the past 20 years” 
(Newmark & De Rugy, 2006). In a sense, they were seizing this opportunity, while the district was in a moment of 
weakness, to secede from the union. 

Similarly, the mayor’s committee seized on this break from business as usual to plan a completely 
redesigned district. Led by educational elites and including feedback from 1500 citizens, they engaged in the process 
of redesign very much akin to the idealized design process explained by Banathy (1992). They acted because they 
realized that the system as it existed prior to Katrina was dead, and they sought to ensure that sufficient thought was 
put into the redesign of the new system. This group stands alone as the only stakeholder group in this study that did 
not make an overt grab for power during the first six months following Katrina. They shared their final plan with the 
community on January 17, 2006 and have since retreated from the limelight as the state and other groups have 
carved up the district and ceased to examine the system as a whole. 

Finally, the state had desired to increase its control over NOPS prior to the storm, and used the power- 
vacuum to carry out its takeover plans. State takeover was not a new idea in Louisiana, as the state attempted to take 
over NOPS in 1960 to prevent federal desegregation orders (Inger, 1969). The traditional argument preventing state 
takeover was that local people were best suited to solve local problems. Now it seemed that local people had more 
serious issues of housing and economic stability to worry about. Conveniently for the state any political 
organizations that might have protested such a move were temporarily unable to mount any opposition. The state 
used this opportunity to amend the pre-existing takeover law and place most of the district under its direct control. 

Findings #2: What basis did stakeholder organizations use to argue for their proposed reforms? 

Equally as important to understanding the process of urban school reform is the rationale each group gave 
for their post-Katrina reforms. The teachers union argued that immediately opening schools would bring back 
population (and help the economy) as well as serve to bring the community together in this time of need. This basis 
looks past efforts to radically alter the pre-existing system and focuses on the need for immediate school reopening. 
This would be expected as the union’s role is protecting the interests of its members, all of whom were without pay 
as long as schools were closed. 

The school board acknowledged that they could not adequately bring the district back, and that schools 
would be back faster, and perhaps improved more readily, with independent leadership. The fact that the board was 
in serious financial straits prior to the storm led them to the position that they were unable to bring the district back 
even to where it had been. Federal charter money and the intense public interest in new reforming their schools 
made new leadership appear as the best route to ensuring the education of New Orleans’ returning children. 

ACSA, in the process of gaining control of 13 schools under its single charter, argued that a community 
run, low-bureaucracy school system would serve students better than the old system. They would be able to manage 
a much smaller budget and staff and avoid much of the mismanagement that plagued the pre-Katrina district. 
Embedded in their argument is a version of the local control argument that says that problems can be solved better 
by those who are closest to them. This argument resonated well in a city that had been ill-served by a centralized 
school board and had been left to drown by FEMA’s bungling antics in response to Katrina. 

The mayor’s committee argued that a broad base of input and careful, centralized planning was needed as 
part of a comprehensive plan to overhaul the district. Committee members knew that this was a once-in-a-century 
event that provided a unique opportunity to intervene in the downward spiral of the New Orleans Public Schools. 
This rationale resonates with Kurt Lewin’s (1951) notion of freezing and unfreezing in organizational change. It 
could be argued that Katrina’s aftermath was a period when the relatively rigid structures of the district were 
malleable and that soon after the return of the population and the re-opening of schools, the district would refreeze 
into a new rigid structure. Thus, making the most of this opportunity was essential to the mayor’s group. 

Finally, the state of Louisiana had always viewed New Orleans schools as an embarrassment and an 
economic hindrance. Now was the time for the state to exercise its constitutional power to take direct control of the 
district. This was a move that would put the burden of running the troubled system on the state, but would move the 
management of the system away from the local political infighting that they assumed to be a cause of the problems. 
Embedded here is that a rational, resourceful government can manage affairs better than locals who are blinded by 
everyday affairs, which can, admittedly, get pretty blurry in the city of New Orleans. This argument, of course, is 
not new- and was made often in the early part of the 20 th century to eliminate inefficiency from school systems 
(Tyack & Cuban, 1995). 
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Findings #3: What successes/failures have groups had in bringing their educational vision to fruition? 

There has been a great imbalance between what groups have been successful in getting their vision 
implemented, much of this having to do with political power and the ability to compose a reform message that was 
pleasurable to the ears of decision makers. The union and the board had no credibility due to their past failures in 
reforming the district. It appears that the board’s mission may have been merely to divest itself of much of its 
authority in the district, in which case they were quite successful at implementing its mission. This does not, 
however, give much power to their educational vision, if their even was one. The union continued to harass the 
actions of others, but not engage in any sort of helpful dialogue with the district that has made other urban reforms 
successful (Urbanski, 2000). Thus, they have been pushed aside and serve as representatives, but not an important 
force in guiding the direction of the district. 

The Mayor’s committee brought national media attention and harnessed the support of recognized educational 
experts in formulating a massively revised plan for NOPS. But, they had no formal power and since the state take 
over, little of their plan has been implemented. As the state looks for guidance in running a school system (Nelson, 
2006), the committee’s “educational network model” may come back, but as of this writing, it has been put on the 
shelf. 

The state and ACSA have had the most success in getting their educational vision for reform implemented. 
ACSA has done this by being the first major chunk of schools to be up and running and doing a reasonably good job 
of educating 1 in 5 of New Orleans students. The state has exercised its constitutional authority and controls the 
largest number of schools, but what exactly the state’s influence will be is largely uncertain. The opening of schools 
in Sept. 2006 was fraught with problems in the state-run schools (Saulny, 2006; Ritea, 2006e) and state-appointed 
RSD superintendent Robin Jarvis announced her resignation in May 2007 after about a year on the job. 

Conclusions 

In this case of urban school reform initiated by a natural disaster, stakeholder organizations reacted in a 
number of ways, from divesting of authority, to creating sub-districts, to engaging in dialogue about what a new 
system might look like. In this case, it appears that most of those who put forth visions of a new system have gained 
some measure of control in the post-Katrina era educational landscape. In other words, those who successfully 
espoused a particular cultural change within the district where often successful in achieving some of the structural 
changes that they sought. The union and the school board did not propose significant alterations to the practice or 
function of education in the city, and have thus been pushed to the margins of the educational community. 

ACSA, the state, and the mayor’s committee all have put forth visions of a new educational system and all 
have found some purchase in New Orleans. ACSA’s regional charter system represents an experiment as an urban 
K12 charter school district and has been sanctioned by the state to continue its operations as long as academic 
achievement does not falter. The state has taken over one-hundred buildings (half of which are empty) and is 
granting more charters as well as running some schools itself. As of November 2006, the district is approaching half 
of its pre-Katrina enrollment: 36% of the students are enrolled in independent charter schools, 18% in the ACSA 
charter network, 35% in the state-run RSD, and 11% in the few remaining district-run schools (see Figures 1 and 2). 
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Sept. 21 Enrollments In New Orleans Public Schools 




19 % 



Figure 1 Enrollment in New Orleans Public Schools September 21, 2006 



November 26 School Enrollments In New Orleans Public Schools 



11 % 




18 % 



Figure 2 Enrollment in New Orleans Public Schools November 26, 2006 
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What this patchwork quilt of educational options may tell us is that different theories of reform and 
different ideologies might exist within the same city, even the same district- if the “district” is conceived broadly 
enough. The opportunity to make substantive cultural changes in urban classrooms may also require the ability to 
carve out some space within the bureaucratic structure to enact those classroom and school level changes. This 
finding may support the type of loose-coupling (Weick, 1976) between districts and individual schools that allows 
both district-level support and freedom for schools to experiment. This is a structure that has been popular with 
urban reformers promoting magnet schools and vouchers in the past (Dougherty, 2004). It should be noted that such 
a structure was recommended by the mayor’s BNOB-ED committee in their “educational network model.” Time 
will tell if such a district model will be implemented in New Orleans and further research is required to determine if 
the structural changes already made will indeed lead to the cultural changes which have been proposed. To be 
certain, the citizens of New Orleans are watching this experiment closely and the preliminary results of nation’s 
largest experiment in urban school choice will be made public when school report cards come out in summer 2007. 
Until then, we can just continue to watch as events unfold. 
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Conflicts in Human-Computer Interactions: A Framework for Designing 
Effective Message Exchange between Humans and Pedagogical Agents 
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Abstract 

We present guidelines for improving real-time interactions between learners and pedagogical agents. We argue that 
learners treat virtual characters as human counterparts and are often frustrated with pedagogical agents’ interactional 
capabilities. To improve the way agents interact with learners, we draw on guidelines that have proven to be 
effective in human-human communication literature and apply those to the design of pedagogical agents. 

Researchers have long predicted the integration of artificial intelligence characters in curricula to assist in 
student learning. Such characters have been termed pedagogical agents and can be defined as human-like virtual 
characters employed in electronic learning environments to serve various instructional goals (Baylor, 2002). Even 
though advances in hardware and software have enabled the use of pedagogical agents in educational circles, 
conflicts arise when students hold real-time conversations with pedagogical agents. Most notably, learners become 
frustrated with pedagogical agents and frequently express negative feelings when a virtual character does not 
respond in the way that learners expect. The following quote from a participant in a study conducted by Doering and 
Veletsianos (2007) illuminates such feelings: 

I hated Joan or whatever the super-agent lady was called. She asked me at one point, 'Are you 
testing me?' like we were going to have some sort of a confrontation or something. I've never 
wanted to hurt a digital person before! 

Given that the assimilation of virtual characters in electronic learning environments is expected to continue 
(e.g. Federation of American Scientists, 2006), an investigation of how to foster effective interaction between 
humans and agents appears to be worthwhile. Even though researchers have already made disjointed attempts to 
enhance the interactional capabilities of agents such that they appear cooperative, polite, friendly, and 
knowledgeable, this paper proposes an extensive framework for the design of agents’ interactional capabilities. This 
framework is derived from a model of effective human-human communications and is applied to the design of 
conversational systems. To the best of our knowledge this framework is the first instance of a well-rounded 
approach to the design of pedagogical agents’ interactional capabilities. 

In this paper, we will provide a review of pedagogical agents, explain the conflicts that arise when learners 
interact with pedagogical agents, and provide a framework to reflect upon and guide the design of effective agent- 
learner conversations. 



Pedagogical Agents: A Review 

Pedagogical agent research has been expansive. Researchers from varied fields such as educational 
technology and psychology, human-computer interaction, computer science, design, and linguistics have examined 
aspects of pedagogical agents with regards to numerous facets of interest including learning, teaching, and 
cooperation. Such research has so far been predominantly experimental and quasi-experimental (Mahmood and 
Ferneley, 2006), striving to discover cause and effect relationships, and measure the impact of various virtual 
character features (e.g., instructional role, gender, voice) on a number of quantitative variables (e.g., performance, 
perceptions of agent traits). It is also important to note that pedagogical agent integration transcends the content-area 
in which pedagogical agents are used. For example, pedagogical agents have been utilized in the study of economics 
(Baylor & Ryu, 2003), physics (Lester et al, 1997), nanotechnology (Hershey-Dirkin, Mishra, & Altermatt, 2005), 
computer literacy (Graesser et al., 1999), and mathematics (Baylor, Shen, & Warren, 2004). 

One of the theoretical arguments used to support the integration of pedagogical agents in education is that 
they can enact socio-cultural aspects of learning (Gulz, 2004), in effect enhancing social interactions (Vygotsky, 
1978) between learners, teachers, and computers. In other words, previous work suggests that social interaction 
between learners and between teachers and learners may enhance learning, and this work is often applied to the use 
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of pedagogical agents. Previous studies have presented overwhelming experimental evidence showing that humans 
treat media as if they were human, proposing that our responses to media are inherently social in nature (Reeves & 
Nass, 1996). This evidence has been replicated in both phenomenological settings (Veletsianos, 2006a) and 
longitudinal qualitative studies (Doering, Veletsianos, & Yerasimou, in press). For example, one of the research 
studies presented by Reeves and Nass (1996, p. 53-58) begins with the understanding that in real life people think 
better of people who praise or flatter them than people who make no evaluation. Transforming this psychological 
finding to a human-computer interaction context, it implies that people will like the computer more when it flatters 
them about their work than when it does not say anything at all. Reeves and Nass tested this hypothesis by devising 
an experiment where the computer either flattered or said nothing to participants after the completion of a task. 
Results showed that participants who were flattered thought that they did better on the task and liked the computer 
more than participants who were not flattered, even in the cases were the flattery was insincere. 

These results suggest that if humans treat computers and media as if they are human, social interaction 
between humans and computers is expected to approximate social interaction between humans and humans. 
Additionally, if the computer is represented as a human-like character able to dynamically converse and interact in 
real-time with the user the distinction between human-human and human-computer interaction may become even 
blurrier (Veletsianos, 2006b). Holtgraves et al. (in press) provide support for such a hypothesis by presenting 
experimental evidence indicating that participants viewed conversational software systems as encompassing human- 
like personalities. For example, a character who responded quickly was perceived to be more conscientious and 
extraverted than a character who did not respond as quickly, leading the authors to argue that, “perceptions of the 
hot were influenced by communication variables that have been demonstrated to influence perceptions of human 
communicators” (p. 10). This finding further implies that humans would expect their interactions with media and 
conversational systems to abide by human norms. It follows that an examination of effective communication 
between humans and humans may be beneficial in informing designers with regards to effective communication 
between humans and computers. 

Human-Computer Interactions: Defining the Conflict 

Conflicts are ubiquitous. From a misunderstanding between colleagues, to an argument between spouses, to 
an internal struggle, to differing opinions between countries, everyone has at one point or another experienced a 
conflict in their life. This outlook of conflicts is exemplified by Deutsch (1973) who notes that a conflict exists 
whenever incompatible activities occur. An incompatible activity is one that “prevents, blocks, or interferes with the 
occurrence or effectiveness” of a second activity. 

Even though Johnson and Johnson (2003) are concerned with how to resolve intrapersonal, interpersonal, 
and intergroup conflicts, incompatible activities occur when humans interact with computers. The Johnson brothers 
(2006, pp. 370-371) define a conflict of interest as an interpersonal conflict that exists “when the actions of one 
person attempting to maximize his or her benefits prevent, block, interfere with, injure or in some way make les 
effective the actions of another person attempting to maximize his or her benefits” This definition can be extended 
to human-computer interaction and defined broadly as follows: An HCI conflict occurs when the actions of the 
computer trying to achieve goals prevent, block or interfere with a person’s attempt to achieve goals. More 
specifically for the purposes of this paper, an HCI conflict can be operationalized as a conflict that occurs when the 
actions of a pedagogical agent trying to achieve educational goals prevent, block, or interfere with the learner’s 
attempt to achieve educational goals. 

One of the issues that keeps resurfacing in the pedagogical agent literature is that the interactional 
capabilities of pedagogical agents and the technology behind intelligent tutoring systems are not yet capable to 
enable effective human-computer interaction (Doering, Veletsianos, and Yerasimou, in press; Gulz, 2004). For 
example, when agents do not “know” the answer to a question, they may respond in a multitude of ways including 
avoiding to answer questions, mocking the user, questioning the user’s intentions, and respond by using generalized 
and out-of-context comments. Additionally, valid agent responses may not conform to human-human conversation 
norms, oftentimes appearing to be defensive or even aggressive. 

Improving Agent-Student Interactions: A Conflict Resolution Approach 

Johnson and Johnson (2006) note that there are two ways to manage conflicts - constructively or 
destructively. Resolving conflicts constructively leads to numerous beneficial outcomes including strengthening 
liking, respect and mutual trust, while resolving conflicts destructively leads to anger, resentment and distrust. The 
Teaching Students to be Peacemakers (TSP) program proposed by Johnson and Johnson (2003) is based on the 
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understanding that conflicts exist in schools and that without training, students tend to resolve conflicts 
destructively. The TSP program provides an extensive framework to train students to solve conflicts constructively. 
The program has been widely implemented and researched and it appears to be an effective intervention for the 
constructive resolution of conflicts in school settings (Johnson and Johnson, 2001). In the TSP program, students are 
required to work cooperatively, negotiate with each other, mediate conflicts, and problem solve in order to reach a 
constructive resolution of the conflict. These actions however require students to effectively communicate with each 
other. Effective communication is defined as the situation “when the receiver interprets the sender’s message in the 
same way the sender intended it” (Johnson and Johnson, 2006, p. 133) and it is paramount for the smooth 
functioning of groups. Given that implementations of pedagogical agents in the classroom often require the 
collaboration between agents and students, effective communication between students and agents is vital if we our 
goal is to attain meaningful and worthwhile agent-student interactions. 

Effective communication between individuals appears to be one of the cornerstones of managing conflicts 
constructively and as such Johnson and Johnson (2006) present guidelines on how individuals can engage in 
effective communication by sending effective messages (Table 1). 



Table 1: Guidelines for sending messages effectively 



Guideline Description 


1 


Clearly own your message. 


2 


Establish your credibility. 


3 


Make your messages complete and specific. 


4 


Make your verbal and nonverbal messages congruent. 


5 


Be redundant. 


6 


Ask for feedback. 


7 


Make the message appropriate to the receiver’s frame of reference. 


8 


Describe your feelings by name, action, or figure of speech. 


9 


Describe other’s behavior without evaluating or interpreting. 



Therefore, if humans treat media as humans, the theory implies that these guidelines should also work in 
the design of pedagogical agents. Specifically, these guidelines are important in the design of messages to be 
delivered from the agent to the learner because they have been shown to be valid in human-human communication. 
Hence, we propose that these guidelines should be examined more closely in the design of pedagogical agent 
responses. In the paragraphs that follow, we will examine each guideline from the agent’s perspective and explain 
how it can be used in the context of human-computer interactions. 

Clearly own your message 

In sending effective messages, students are encouraged to take ownership and responsibility of their 
statements by using first personal pronouns such as “I” and “my.” The use of phrases such as “some people believe 
that” or “most people note that” may indicate a refusal to acknowledge ownership and responsibility of a comment. 
Such comments appear too generalized, without indicating to the receiver the source of such messages. Applying 
this guideline to pedagogical agent message design implies that the agent should establish ownership of the message 
and take responsibility for the response. In other words, the agent needs to be clear in that the comments (s)he makes 
come from him/herself. 

Establish your credibility 

Credibility refers to the perceptions of the receiver regarding the sender’s messages trustworthiness. 
According to Johnson and Johnson (2006, p. 136) a credible sender is one who is perceived to be “(a) reliable as an 
information source, (b) motivated to tell the truth, (c) warm and friendly, (d) trustworthy, (e) in possession of 
expertise, and (f) dynamic.” In human-computer interaction contexts this implies that the human should perceive the 
computer to be trustworthy. This is a delicate guideline that has been hotly debated in the HCI literature (e.g. Fogg 
and Tseng, 1999), with a set of related website credibility guidelines proposed by the Stanford Persuasive 
Technology Lab. In line with the credibility guidelines proposed by Johnson and Johnson, credibility can be 
established when the learner perceives the agent to fulfill the six characteristics presented above. It is important to 
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note that these characteristics can be fulfilled not only by modifying message design (e.g. appearing warm and 
friendly by affectionately greeting the learner) but also by adjusting the looks of the agent (e.g. a formally dressed 
professor may appear to be more knowledgeable than a green parrot). 

Make your messages complete and specific 

In communicating about ideas, feelings, and opinions, students should use statements that are clear and 
inclusive of all necessary information needed by the receiver to understand what the sender is trying to 
communicate. Completeness and specificity are important but when we communicate with others we usually do not 
indicate our frame of reference, assumptions, or the leap of thinking we are making. For effective communication to 
take place, our statements should be expansive and, ideally, should include what we otherwise would have failed to 
state. In human-computer interaction settings this guideline implies that the agent should give specific and complete 
responses. Such responses should be specific in the sense that they should not be plagued with assumptions about 
the learner or about the knowledge of the learner. Additionally they should encompass the frame of reference from 
the agent’s point of view. 

Make your verbal and nonverbal messages congruen 

In human-human communication participants interact with both verbal and non-verbal messages. 
Frequently, these messages are congruent - verbal messages match nonverbal messages. For example, a “Hi, my 
friend” accompanied by a smile and a pat on the back indicates the greeter’s positive emotions. If the “Hi, my 
friend” was followed by a sneer on the greeter’s face, the meaning of the verbal information might change because 
of the co-existence of contradictory messages. Therefore, in human-human conversations verbal and nonverbal 
messages should be congruent so as to avoid misunderstandings between people. This guideline is especially 
important in human-computer interactions. It implies that verbal responses should match the agent’s nonverbal 
messages so as to avoid misunderstandings between student and agent. At present, most pedagogical agents use a 
neutral text-to-speech computer generated voice that lacks voice inflection making verbal messages flat and possibly 
unclear. Additionally, we need to examine how to generate dynamic facial expressions that vary with the agents’ 
verbal messages. An important obstacle that still needs to be overcome is the creation of algorithms that enable 
agents to adapt their n th verbal and nonverbal messages depending on their verbal and nonverbal n-l th response. 

Be redundant 

Redundancy in effective communication implies sending the same message more than once and using more 
than one channel of communication to convey a message. For example, when a student is attempting to explain to 
another student why he felt cheated while they were playing chess, he could use a chess board to re-enact the 
conflict of interest. The same guideline extends to human-computer interactions. Pedagogical agents can utilize a 
database of images to present information and explanations to learners such that student understanding can be 
advanced. 

Ask for feedback 

Students should ask for feedback of how the receiver perceives, understands, and interprets their messages. 
Feedback can therefore facilitate a conversation because misunderstandings can be prevented. In the same way, 
pedagogical agents can ask for feedback concerning the response s(he) has given to a learner. This feedback can 
serve two purposes: First, the computer can log the user’s response and designers can modify the agent database 
according to such feedback, and second, the agent can adapt his/her response according to how the message was 
received from the learner. The challenge with this guideline lays on the fact that artificial intelligence engines need 
to evolve beyond the pattern and keyword matching approach and scripted response algorithms to sophisticated 
evolutionary models of interaction where meaningful responses are formed dynamically and intuitively. In 
educational technology for example, researchers have expressed their dissatisfaction with the current state of 
artificial intelligence (Gulz, 2004) and noted the difficulties of exploring advanced conversational techniques in 
student-agent interactions. 

Make the message appropriate to the receiver ’s frame of reference 

The same information should be explained differently to students of differing levels of expertise. For 
example, relativity theory should be explained differently to 3 rd graders than to college sophomores because they 
have a different frame of reference and knowledge. This guideline implies that agents should be able to dynamically 
alter their responses and information depending on who they are interacting with. Previous research has shown that 
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novices understand and perceive information differently than experts, and this understanding should be taken into 
conideration when designing pedagogical agents. 

Describe your feelings by name, action, or figure of speech 

Johnson and Johnson (2006) note that it is important that descriptions of feelings should be clear and 
unambiguous. To this end, they suggest that feelings could be described by name (I am happy), action (I feel like 
smiling) or by figure of thing (I am flying with joy). In the same way, I recommend that agents should be descriptive 
about their “feelings” when interacting with learners. For example, agents can congratulate learners when they 
complete a task by unambiguously stating, “I am happy that you completed module 8 on nuclear thermodynamics.” 

Describe other ’s behavior without evaluating or interpreting 

When students are attempting to resolve conflicts it is important that their comments are descriptive rather 
than evaluative. For example, it would be preferable for a student to say “You stood in front of me in line” rather 
than saying “You were self-centered by taking my place in line.” In the same way, pedagogical agents should not 
make descriptive rather than evaluative comments. For example, consider the case were a learner spends 10 minutes 
on a module and only scores 8/20. The agent can give the following descriptive feedback: “You scored 8/20 by 
spending 10 minutes on the module. Spending more time on this material will help in raising your score.” An 
evaluative feedback would sound more judgmental and it could be similar to the following: “You only spend 10 
minutes on this module and scored 8/20. You can raise your score next time by spending just a little bit more time 
on the task!” 



Conclusion 

The design framework we presented in this paper can be used to investigate, research, and enhance the 
interactional capabilities of pedagogical agents. To date, ongoing research completed as part of our research agenda 
indicates that the framework proposed within this paper is an effective first step into a well-rounded approach to 
enhancing learner-agent interactions. 



33 




References 



Baylor, A.L. (2002). Expanding Preservice Teachers’ Metacognitive Awareness of Instructional Planning Through 
Pedagogical Agents. Educational Technology Research & Development, 50(2), 5-22. 

Baylor, A. L., and Ryu, J. (2003). The Effects of Image and Animation in Enhancing Pedagogical Agent Persona. 
Journal of Educational Computing Research, 28(4), 373-394. 

Baylor, A. L., Shen, E., and Warren, D. (2004). Supporting learners with math anxiety’: The impact of pedagogical 
agent emotional and motivational support. Paper presented at the International Conference on Intelligent 
Tutoring Systems, Maceio, Brazil. Retrieved May 1, 2005 from the World Wide Web: 
http://ritl.fsu.edu/papersAV7_baylor_revised.pdf 

Doering, A., Veletsianos, G., & Yerasimou, T. (in press). Conversational Agents and their Longitudinal Affordances 
on Communication and Interaction. Journal of Interactive Learning Research, 19(2). 

Doering, A., & Veletsianos, G. (2007). Multi-Scaffolding Learning Environment: An Analysis of Scaffolding and 
Its Impact on Cognitive Load and Problem-Solving Ability. Journal of Educational Computing Research, 
37(2), 107-129. 

Federation of American Scientists (2006). Harnessing the power of video games for learning. Report accessed on 
November 4, 2006 from 

http://fas.org/gamesummit/Resources/Summit%20on%20Educational%20Games.pdf. 

Fogg, BJ, and Tseng, H. (1999). The elements of computer credibility. Paper presented at the annual Meeting of the 
Computer Human Interaction (CHI) conference (Pittsburgh, PA). 

Graesser, A.C., Wiemer-Hastings, K., Wiemer-Hastings, P., Kreuz, R., and the Tutoring Research Group (1999). 
AutoTutor: A simulation of a human tutor. Journal of Cognitive Systems Research, 1, 35-51. 

Gulz, A. (2004). Benefits of virtual characters in computer based learning environments: Claims and evidence. 
International Journal of Artificial Intelligence in Education, 14, 313-334. 

Hershey-Dirkin, K., Mishra, P. and Altermatt, E. (2005). All or Nothing: Levels of Sociability of a Pedagogical 
Software Agent and its Impact on Student Perceptions and Learning. Journal of Educational Multimedia 
and Hypermedia. 14(2), 113-127. 

Holtgraves, T., Ross, S., Weywadt, C., and Han, T. (in press). Perceiving artificial social agents. Computers and 
Human Behavior. 

Johnson, D., & Johnson, F. (2006). Joining together: Group theory and group skills. Boston: Allyn & Bacon. 

Johnson, D., & Johnson, R. (2001). Teaching Students To Be Peacemakers: A Meta-Analysis. Paper presented at the 
Annual Meeting of the American Educational Research Association (Seattle, WA, April 10-14, 2001). 

Johnson, D., & Johnson, R. (2003). Teaching students to be peacemakers (3 rd Ed.). Edina, MN: Interaction Book 
Company. 

Jonassen, D., Campbell, J., and Davidson, M. (1994). Learning with media: Restructuring the debate. Educational 
Technology Research and Development, 42(2), 31-39. 

Lester, J., Converse, S., Kahler, S., Barlow, T., Stone, B., and Bhogal, R. (1997). The persona effect: Affective 

impact of animated pedagogical agents. Proceedings of Human Computer Interaction (CHI), Association 
for Computing Machinery (ACM), 1997, 359-366. 

Mahmood, A., & Ferneley, E. (2006). Embodied agents in e-learning environments: An exploratory case study. 
Journal of Interactive Learning Research, 17(2), 143-162. 

Reeves, B., and Nass, C. I. (1996). The media equation: How people treat computers, television, and new media as 
real people and places. Cambridge: Cambridge University Press/CSLl. 

Veletsianos, G. (2006a) Conversing with intelligent agents: A phenomenological exploration of communication with 
digital entities. Unpublished manuscript. University of Minnesota. To be presented at the AERA 2007 
conference in Chicago, 1L. 

Veletsianos, G. (2006b). Contextual Pedagogical Agents: Stereotypes and First impressions and their Impact on 
Student Learning and Perceptions of Agent Persona. Unpublished Master’s thesis. University of 
Minnesota. 

Vygotsky, L.S. (1978). Mind in Society’: The development of higher psychological processes. Cambridge, MA: 
Harvard University Press. 



34 




Collaborative and Sustainable Instructional Design Model for Service 

Learning 

Brian R. Belland 
bbelland@purdue.edu 

Vanitha Vaithinathan 
vanitha@purdue . edu 

Belen Garcia 
garcia20@purdue.edu 

Department of Curriculum and Instruction 
Purdue University 
3134 BRNG 
100 N. University St. 

West Lafayette, IN 47907 



Wen-Hao Huang 
wdhuang@uiuc .edu 

Department of Human Resource Education 
University of Illinois, Urbana-Champaign 
35 1 Education Building 
1310 S. 6 th St. 

Champaign, IL 61820 

Abstract 

Service Learning (SL), in which students learn content while performing service in their community, has been used 
increasingly in K-12 and university contexts (Seitsinger, 2005). Service or Learning often dominates “SL courses” 
due to lack of effective front-end analysis and ongoing communication between stakeholders. We created a new 
instructional design model to encourage designers of SL courses to link community needs to learning needs and 
develop communication supports to balance and sustain SL projects. 

Introduction 

Gagne, Briggs and Wager (1988) defined instruction as “a deliberately arranged set of external events 
designed to support internal learning processes” (p. 11), and noted that the first step in the design of instruction is 
the definition of intended learning outcomes in terms of measurable objectives. But what happens when traditional, 
performance -based learning outcomes are not the only intended outcomes of instruction? In this paper we describe a 
new instructional design model that guides designers to consider two sets of needs (and thus outcomesj-Learning 
and Community-when designing Service Learning courses. 

Service Learning: A Definition 

Service Learning is a pedagogical approach in which students complete projects that render service to 
specific communities in order to meet learning needs (Billig, 2002). Service Learning has been used increasingly in 
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university (Buchanan, Baldwin, & Rudisill, 2002; Leh, 2005) and K-12 (Skinner & Chapman, 2000) settings in large 
part because it is said to produce social gains such as increased self-confidence and desire to be involved in the 
community (Billig; Manley, Buffa, Dube & Reed, 2006; Skinner & Chapman) and academic gains such as higher 
grades and scores on standardized tests (Billig). 

Service Learning is distinguishable from community service because its projects benefit equally students 
and the communities the projects serve. Service Learning is widely used in teacher education programs, where pre- 
service teachers often mentor and tutor children and adolescents at local schools (Buchanan et ah, 2002; Ryan & 
Callahan, 2002). Service Learning is also used to help students learn about history; in one such project, college 
students worked with high school students to research historically important but largely forgotten neighborhoods 
and take oral histories from residents (Manley et ah, 2006). In each case, both communities (K-12 students) and 
students (pre-service teachers or other college students) benefited from the project. Pre-service teachers honed 
teaching techniques while helping K-12 students learn and grow (Buchanan et ah; Ryan & Callahan). History 
students improved their historical record taking abilities while also being able to help K-12 students understand an 
important slice of history (Manley et ah). 



Why a New Model? 

Designing Service Learning courses is fundamentally different from designing other courses because, in 
designing Service Learning courses, two sets of needs must be considered: community needs and learning needs. In 
order to avoid creating a Service learning course (where service dominates learning) or a service Learning course 
(where learning dominates service), it is important to consider both sets of needs at the same time in the design 
process (Tholecken, Clark, & Tschirch, 2004). For the seasoned designer, this may not be a challenge, as 
experienced designers do not follow instructional design models religiously (Gustafson, 2002). However, most 
designers of Service Learning courses are professors and teachers in various content areas, and are not seasoned 
instructional designers or experienced designers of Service Learning courses (Billig, 2002). When presented the 
challenge of designing a Service Learning course for the first time, designers of Service Learning courses may look 
to instructional design models for guidance. Opening up the typical instructional systems design model they would 
notice the first step towards designing their course is needs assessment and analysis (Gustafson & Branch, 2002; 
Kaufman & Thiagarajan, 1987). They would learn that needs assessment is identifying the gaps between what 
students are able to do, and what they should be able to do, and that needs analysis is determining which of the gaps 
can be addressed through instruction. They would also learn how to perform these steps. However, this newfound 
knowledge would not necessarily help designers of Service Learning courses design courses that balance Learning 
and Community needs because there is no provision in existing instructional design models for considering 
community service needs during needs analysis (Gustafson & Branch; Kaufman & Thigarajan). 

Purpose of this Paper 

Instructional design (ID) models provide guidance for designing and developing instructional materials 
and/or units. Existing ID models do not provide sufficient guidance for designing Service Learning courses 
primarily because two sets of needs should drive Service Learning course design: learning and community needs. 
Based on our experience designing and developing Service Learning courses, we created the Collaborative and 
Sustainable Instructional Design model for Service Learning (CSIDSL) to help designers design and develop 
Service Learning courses while keeping the two complementary sets of needs (learning and community) in mind and 
involving community and students in the design process to promote effective communication and sustainability. 

Guiding Assumptions in Development of the Model 

Several assumptions guided our development of the CSIDSL model. Unique characteristics of the model 
along with the corresponding assumptions are outlined in Table 1. 
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Table 1 

Guiding Assumptions and Characteristics of CSIDSL 



Characteristic 

Collaborative 

Sustainable 

Community/Learning 
Needs Analysis 

Continuous improvement 



Guiding Assumptions 

Students who participate in a Service Learning project’s design will feel a stake and be 
motivated to excel in the project (Billig, 2002; Swan, 2006; Werner, Voce, Openshaw, & 
Simons, 2002) 

Community stakeholders who participate in a Service Learning project’s design will be 
more likely to actively participate in the project (Yoder, 2006) 

Service Learning courses are difficult and time-consuming to design (O’ Quin, Bulot, & 
Johnson, 2005); thus, it makes sense to develop long-term relationships with community 
agencies so that one term’s Service Learning project can be either continued or built upon 
in the following term (Yoder, 2006) 

Designers must consider community and learning needs throughout the design process so 
as not to let service or learning dominate Service Learning (Burns, 1998; Manley et al., 
2006) 

Because Service Learning projects involve multiple stakeholders and run for many weeks, 
all formative evaluation and revision of instructional material cannot be accomplished 
before the start of the course. Therefore, formative evaluation should be performed during 
the term, and revision should either be performed during the term if possible, or after term 
end to prepare for the next term. 
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The Model 



CSIDSL consists of five iterative stages: 



Figure 1. CIDSL. 




Designers must start at the center triangle, and then can proceed to other stages. In the following section we 
provide a brief description and illustration of how to perform each stage. 

Define Community’ Needs, Student Needs, and Constraints 

In order to design a project that meets both student and community needs, it is necessary to first know each 
set of needs. In this stage, designers identify student needs. Then they must specify a community agency to serve, 
and define its needs. For example, if a group of preservice teachers needs to learn a new reading instruction 
technique, an appropriate community may be kindergartners. Once an appropriate community is found, a 
participating agency, such as a specific kindergarten, can be identified. Shared needs (illustrated by the overlap 
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between student needs and community needs) are tasks or knowledge students need to learn but the community 
agency needs to have done. 

The designer must first establish the learning needs. If the Service Learning course to be developed is a 
graduate course in Department X, a good starting point is to interview students in Department X to see what learning 
needs they have that are not being met by the current curriculum. The learning needs the dictate a body of possible 
community agencies to serve. Because not all community agencies may be able and willing to be involved in a 
Service Learning project, selecting a cooperating community agency will involve communicating with the different 
agencies to determine a best fit. After establishing an appropriate community agency to serve, a designer must 
contact the agency to explore where its needs may overlap with the learning needs. Pretests and interviews can help 
designers identify constraints, or factors that limit designers’ ability to fully address all shared needs, such as amount 
of time available and students’ existing competency. 

Identify Instructional Goals and Design Assessment 

Instructional goals in the context of a Service Learning course are no different from instructional goals in 
the context of other types of instruction: they remain what the instruction should help students learn. For example, if 
preservice teachers needed to know how to use brand X reading instruction and brand Y algebra instruction 
methods, and the community (i.e., kindergartners) needed to learn to read, then the instructional goal should be 
based off the shared need: reading instruction. The resulting instructional goal would be that students be able to use 
the brand X reading instruction method to teach kindergartners to read. 

Assessment should serve to measure the extent to which students meet the unit’s instructional goals. Most 
service-learning courses include both content (e.g., students will be able to apply X principle in Y situations) and 
affective (e.g., students will choose to become involved in the community in their chosen field) instructional goals. 
One type of assessment that can measure students’ attainment of both such goal types is guided reflection, in which 
students are given prompts to write reflections about their experiences both during and at the end of the project 
(Hatcher & Bringle, 1997; Yoder, 2006). In such experiences, students can articulate both what they accomplished 
and how the experience impacted their beliefs. 

Shared needs dictate instructional goals. When creating instructional goals, designers should be careful to 
describe what learners will be able to do at the end of the 

Service Learning course. Designers should create measurements that assess both affective and content instructional 
objectives through student reflection on their experiences during the service-learning project. Such a measurement 
would include prompts that guide students to consider not only what they did during the unit and how well they did 
it, but also the impact of the project on the community (Hatcher & Bringle, 1997). 

Determine Project Type, Scope, and Sustainability 

The largest student participation in the design process takes place in this stage, as students work with 
instructors to select a motivating project. When determining project type and scope, designers should also consider 
sustainability, or the extent to which a Service Learning project can be sustained beyond the current semester’s 
course. 

Instructional goals, sustainability considerations, and student interests dictate project type. Project scope is 
determined by constraints, such as students’ existing competencies or amount of time available. For example, the 
community agency may need an attractive web site, but if students in a web design class have no prior knowledge of 
Flash, and the unit lasts one week, then constraints dictate that a WYSIWYG web design program (e.g., Microsoft 
FrontPage) be used in the community service project. Given the large time investment in the creation of a Service 
Learning course, the sustainability of a Service Learning project in future semesters should also drive the 
determination of project type and scope. 

Develop and Implement Support 

Constraints help designers flag tasks that should be supported. For example, student communication with 
the partnering community agency, while essential to the success of a Service Learning project, can be difficult, 
especially when the agency and the school are in different communities. Project management can also be 
challenging for students. In order to ensure the success of the Service Learning project, designers must support these 
and other activities. 
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Designers can develop scaffolds that support students’ time management, communication with the 
cooperating community agency, and task performance (Wood, Bruner, & Ross, 1976). To help ensure sustainability 
of the Service Learning project and ease of communication among students and community agencies, many of these 
scaffolds can be part of a computer-based system into which students can type such information as results of 
research and design specifications, and to which the instructor, other members of the student team, and partnering 
community agencies would have access (Kyza & Edelson, 2005). 

Formative Evaluation 

At each stage, designers should conduct formative evaluation to ensure that (a) shared needs are being met, 
(b) service does not dominate learning (and vice versa), and (c) communication channels are optimized for the 
success and sustainability of the project. 

Designers can conduct expert reviews of both content and design, but the traditional steps of one-on-one 
and small group evaluations may not work optimally with instruction developed using the CSIDSL. For example, 
having one or more students go through the instruction may not help determine whether service is dominated by 
learning, or if communication channels are optimized. So formative evaluation in this model should consist of expert 
reviews, and then ongoing assessment during the course of the success of design strategies. While computer-based 
scaffolding could not be changed easily in the middle of the semester to improve communication for that semester’s 
Service Learning course, the information gained could be used to improve communication during subsequent 
semesters. 



Conclusion 

Service Learning is one way to help students learn important content (e.g., local history and history 
research methods) and practice skills (e.g., teaching) in authentic contexts such as local neighborhoods (Manley et 
al., 2006) and K-12 schools (Buchanan et al., 2002; Ryan & Callahan, 2002). Furthermore, Service Learning has 
been shown to produce social benefits such as greater confidence and desire to serve (Billig, 2002; Manley et al.; 
Skinner & Chapman, 2000), as well as academic benefits such as higher grades and test scores (Billig). Flowever, it 
has traditionally been challenging to design Service Learning projects, especially to the novice designers who often 
are responsible for designing such projects. Through the use of CSIDSL, designers can let the confluence of 
community and student needs drive instructional goals. In this ID model, both students and community agencies 
participate in the development of the Service Learning course, and in that way, can feel that they have a personal 
stake in, and be motivated to successfully complete, the project. 
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Choosing a Webconferencing Platform 

Richard A. Berg 



As travel budgets shrink and the need for distance education opportunities rise, Webconferencing has 
become a popular way for organizations to hold meetings, disseminate information, and deliver instruction. 
However, with over 100 different products and service providers available (Woolley 2007), choosing the right 
product or service to meet your needs can be a daunting task. Products and services can be almost free or cost 
thousands of dollars monthly. Simpler products may include the transmission of text or still images while more 
elaborate systems will offer video and voice over IP, text chat, whiteboards, slides with transitions, live software 
demonstrations and a host of other possibilities. Webconferencing can be and is being used for the delivery of 
university courses, holding meetings, training, seminars and collaboration at a distance. The benefits provided by 
webconferencing include reduced travel cost and risk, greater presence or availability of instruction, decreased need 
for special equipment (in most cases), and reduced out of office time. A single university professor teaching via 
webconferencing may be able to reach students at several remote campuses without the need to leave his or her 
“home” campus. A county agent being trained on the use of a new database may be trained via webconferencing 
without the need to ever leave their office. This decreases the costs, risks and hassles of travel while increasing the 
availability of the agent to his or her local clientele. The purpose of this article is not to recommend one product or 
service, but rather to provide assistance that may be useful in determining which product or service will best meet 
your particular needs. 



Beginning the Evaluation Process 



Purchasing a product or service should not be a one-person operation. You will want to assemble an 
investigation team that includes representation of all components of your organization that will have some dealing 
with Webconferencing before and after it is implemented. These people should include clients, faculty members, 
trainers, facilitators, schedulers, technical and support staff, and administrators. While it may be difficult to 
coordinate this team, doing so will pay off in the long run. Depending on your organization’s rules and procedures 
for large purchases, some of these things may already be a part of your pre-purchase procedures. 



The Investigation Team 
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Before your team contacts a sales representative, a thorough needs analysis should be done to determine 
exactly what your organization requires. This may be the delivery of instruction, holding meetings, disseminating 
information, or a combination of these. Depending on your organization, there may also be other needs for 
communicating at a distance. If possible, try to look down the road to see if there are any new needs that may arise. 
After completing the needs analysis, you will be ready to begin asking some questions. 



Action Chart: Choosing a Webconferencing Product or Service 



Process 



Need for a Distance 
Communication 




Solution 





The organization has determined that there is a need for communication across a 
distance. 







The organization has determined that Webconferencing will be one of the distance 



communication mediums. 






H The organization performs a comprehensive needs analysis to determine exactly 
what is needed from a Webconferencing product. 






Formation of 
Investigation Team 




p \ 


I 




Questioning and 
Testing 


p \ 


Narrowing of^N 
the Field 


s- it 


* p \ 


Questioning of 
Current Users 




p \ 


^^Final Decision^ 
on Product 




> p \ 


Purchase and 




Implementation 


p \ 



An Investigation Team is formed. This team will have members from all different 
areas of the organization that are stakeholders in the Webconferencing decision. 
If there are clients outside the organization, they should be represented as well. 



3 



The investigation team determines the possibilities of many different 
webconferencing products and services. 



3 



M Based on their investigations, the team narrows the field to a few strong contenders. 






The investigation team or their agent contacts present users of the products or 
services to determine if those products or services perform as expected. 






The team makes a decision on which product is most likely to meet the needs 
organization and submits a request for purchase. 






M The product is purchased and implemented within the organization. 



3 
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Once your investigation team has been formed, you can begin exploring different areas that will determine 
your decision. The areas and questions that this article explores are not all-inclusive. If there are special needs or 
areas that are unique to your organization, you should include them as part of your investigation phase. 

Some products and services allow free testing and in-house or online demos of what they have to offer. If 
possible, have your team attend a demo, or otherwise obtain a free look at the product to see if it will meet any initial 
needs that were identified in your needs analysis. If a free demo is provided, don’t be afraid to ask questions. 

As with any other technology field, there are many competitive products. Research may indicate whether a 
company is gaining or losing market shares, faltering financially, or about to be bought by another company. 
Products and services from companies that are unstable or are falling in the market may have a very short life span. 

Features and Product Issues 

As the number of products and services grows, the number of capabilities for session leaders also increases. 
Some common features are markup tools, whiteboards, chat, text questions, polls, and application sharing. The tools 
that are available to session leaders should be a good match to what they need to do as part of their sessions. They 
should also be intuitive and easy to use. 

If session leaders need to have collaborative interaction with attendees, there should also be some ability 
for the leaders to share their tools with attendees or otherwise allow them to have some kind of control or input in 
the session. Some products give the session host the ability to “promote” and “demote” session attendees so that 
they have almost all the same abilities as the session host. Other products and services will give the session host the 
ability to share control of one or more tools with attendees. 

The transmission of voice and video through the Webconferencing medium is increasing in popularity. 
There are risks and problems associated with this, however. For the transmission of voice and/or video, users must 
have a large amount of bandwidth available and have a high connection speed. Users on a network with low 
bandwidth and/or with a low connection speed are likely to experience low quality at best and, in some cases, the 
delay, dropping, or freezing of voice and video feeds. Another area of concern is whether the audio transmission is 
full-duplex or half-duplex. If the audio transmission is full-duplex, everyone in a session should be able to speak and 
hear one another at the same time. If the audio is half-duplex, everyone in the session will not be able to speak and 
hear one another at the same time. Half-duplex is somewhat akin to speaking on walkie-talkies, where only one 
person can speak at a time. The investigation team should determine whether or not potential users will have the 
available bandwidth and connection speeds to actually use voice and video features without problems. The team 
should also use the needs analysis to determine if the usefulness of voice or video over IP outweighs the problems 
associated with the additional loads on bandwidth. For example, the use of voice and video over IP for the simple 
purpose of providing a “talking head” is a very poor use of technology. 

If there is to be two-way transmission of voice or video, the costs of equipping attendees with microphones, 
speakers and cameras will enter into the decision. In some organizations, it may not be possible to equip attendees 
with the necessary hardware to obtain the full benefit of attending a session. If there is to be a large portion of the 
session information delivered via live video or audio, there will also be a question of whether or not the session will 
still be accessible to persons with special needs. 

Further, it is important to determine whether or not there are additional charges for the use of video or 
voice over IP. Also, ask if the product or service provides or requires a telephone bridge or other audio service in 
addition to, or instead of voice over IP. 

In some cases, someone may miss a Webconferencing session or may want to review the proceedings. A 
good product or service will usually provide some way to record the session. If recording is available, it is also 
important to determine what options are available with the recording. Different options would include, whether or 
not all screens and annotations are shown in the recording, if audio is included and synched with the video, what 
format the recording is saved in, and whether or not it can be viewed on the Web or downloaded. 
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Product Comparison 



Product 


Minimum 

Connection 

Speed 


Whiteboard 


Markup 

Tools 


Chat/Text 

Messaging 


Polling/ 

Survey 


Desktop/ 

Application 

Sharing 


Tool/ 

Control 

Sharing 


Voice 
over IP 


Video 
over IP 


Archiving/ 

Recording 


WebEx Meeting 
Center 


56K 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Adobe Acrobat 
Connect 


Broadband 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Microsoft Office Live 
Meeting 


56K 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


No 


Yes 


Elluminate Live 
Academic Edition 


28.8K 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Wimba Live 
Classroom 


56K 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 


Yes 



These randomly chosen products all have similar features, but vary in their pricing and pricing structures. 

WebEx Meeting Center (http://www.webex.com/smb/web-meeting-center.html) offers two different monthly subscription options and a pay 
per use option. A live demo and 14-day free trial are available. 

Adobe Acrobat Connect (http://www.adobe.com/products/connect/) offers monthly and annual subscriptions. Free product seminars and a 
15-day free trial are available. 

Microsoft Office Live Meeting (http://main.livemeeting.conV) offers named user, room, shared seat, and per-minute options. A free 60-day 
tnal is available. 

Elluminate Live Academic Edition (http://www.elluminate.com/academic_edition.jsp) offers monthly and annual subscription options. A live 
demo and 2- week free trial are available. 

Wimba Live Classroom (http://www.wimba.com/products/liveclassroom/) offers a monthly subscription. A live demo is available 

Above information current on August 31 , 2007. Listing of the above products does not constitute an endorsement of any of the products over 
one another or of any products not listed. Your investigation team should determine the product or service that will best serve your organiza- 
tion. 

Pricing of different products will vary. Some factors that may contribute to pricing fluctuations within a particular product may include length 
of contract, number of licenses or hours purchased, technical support, and educational or non-profit organizational status 



A determination should also be made as to who will be in charge of recording sessions, where these 
recordings will be stored, and how long they will be stored. Your organization may want to have each session host 
be in charge of making his or her own recordings, storing them and deciding how long they will be kept. On the 
other hand, your organization may want to have one person or department be in charge of making the recordings for 
all sessions, if that is a possibility. 



Implementation and Start-Up Concerns 

If proprietary plugins or add-ons are required, they should be free, safe, and simple to install. On some 
networks, security features will prevent the plugins or add-ons from downloading or installing. There should be a 
Web site or other means to get copies of the plugins so that they can be installed while network security features are 
temporarily disabled. You should also find out if any features that are normally in place, such as internal firewalls or 
pop-up blockers need to be disabled or reconfigured to allow the plugins to function properly. In addition, it is 
important to know if the plugins or add-ons are only required for the session leaders or if they are required for 
everyone taking part in a session. 

If your organization or your clients are using a number of different platforms (PC, Mac, etc.), or browsers 
(Internet Explorer, Firefox, Opera), it will be very important to find a solution that will be usable by everyone the 
new product or service is intended to serve. If you are able to download a free demo of the product or attend a free 
online demonstration, try using a number of different platform and browser configurations to spot any potential 
problems for your intended users. Check to see if the product or service provider has a Web page that lists operating 
system and browser requirements for use, as some products may require the use of a specific browser, operating 
system, or combination of browser and operating system. 
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Some products and services require a high-speed connection, while others are designed to be useable by 
hosts and attendees on a dialup connection. If your potential users are spread across different connection speeds, it 
will be important to find a solution that will allow all of your users to connect without having to travel to a location 
with a higher connection speed. 

If your organization is going to host the product or service, you need to know what size and type of server 
it will require. Once you know that, you will also want to know how hard it will be to configure that server to use 
the product, how hard it will be to implement the product on that server, and how much time your IT staff will need 
to spend maintaining that server. 

If your pool of potential users includes people with special needs, such as blindness or low vision, hearing 
loss or deafness, or motor impairment, it is important to know whether or not the product or service will be 
accessible to them. If the product or service is not readily accessible to people with special needs, it should be 
determined whether or not reasonable accommodations can be made to include these people, providing them with 
full access. The cost and difficulty of making these accommodations should also be taken into consideration. 

A product or service may contain a feature that you do not want your users to have access to for one reason 
or another. Being able to enable or disable certain features at the administrative level should be an option that is 
provided. In other cases, you may want to have a feature added that is not currently available. Find out if that feature 
is planned for a future release or if the product or service provider is willing to add that feature for you. If the 
product or service provider says “yes” to this feature and you decide to choose this product or service provider, 
make sure that the special feature is already in place before signing a contract. 

Using Webconferencing 

There are a lot of products and services available that have all the possible bells and whistles that anyone 
could ever want. Unfortunately, they are sometimes not very intuitive or can be cumbersome to use. Of course, 
“ease of use” will largely depend on how tech-savvy your intended users are. What may seem very difficult for one 
group of potential users may seem quite easy for another group. If your organization has potential users with a wide 
range of skills and abilities, you will want to make certain that the product can be used by individuals with 
somewhat limited skill sets. 

One of the most frustrating things to new technology users is not being able to take all of their present 
materials and use them in a new delivery format. Find out what kinds of materials your potential users are currently 
employing and how easy it will be to integrate them with a new product or service. 

If most people in the organization are new to Webconferencing, there is a good chance that there will be a 
need for post-purchase support, both for users and tech staff. The investigation team should determine what kinds of 
post-purchase support are available, such as training for initial users, phone and email support, and technical 
support. Different products and services will vary on these issues and also on whether or not they charge for these 
services. 



If the vendor does not provide training for users, the organization may need to provide a trainer that will 
help users learn how to use the product. It will be helpful if that trainer also has some knowledge of instructional 
design to assist users who are unaccustomed to this delivery medium. 

Pricing Structures, Session Types, and Scheduling 

The intended use of the product or service will have an effect on what type of pricing structure is chosen. There 
are three basic pricing structures. These are the seat license, the charge per minute, and the room. 

• A seat license pricing structure usually is a monthly or yearly charge that allows one connection to a 
conference. This price is charged automatically, no matter how much or how little that seat is actually used. 
It usually also allows for simultaneous meetings. For example, an organization with 100 seat licenses might 
have two different meetings at the same time, each using 50 seat licenses. 
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• The price per minute is the structure most often used by Webconferencing service providers. This price per 
minute is charged only when the service is actually used. If the service is not being used often, and only by 
a small number of users, the cost will be lower than if it is being used often by a large number of users. 

• The room charge or flat-rate usually allows unlimited use by a pre-specified number of users, and is 
charged on a monthly or yearly basis. This room, however, cannot usually be divided into smaller rooms, 
so the chance for different simultaneous meetings is somewhat limited. 

Before choosing, be certain to know all the limitations and exclusions that apply to the particular pricing 
structure, and what it costs to expand the service so that a larger number of users can be accommodated. As 
competition in the field grows, there are also more pricing structures being developed, so check to see if the provider 
has other pricing plans that may be more accommodating to your organization’s needs. 

Scheduling and setting up meetings varies greatly among different products and services. The investigation 
team should find out how easy it is to schedule and set up a meeting. In some cases, a person may need to be hired to 
take care of all the scheduling and setup of sessions, especially if there will be a large number of different sessions 
occurring simultaneously. In some organizations, users set up their own sessions, while in others everyone has to go 
through a scheduler. It will be important that the scheduling features can be set up to meet your organization’s 
needs. 

Some products and services allow sessions to be set up in advance, while others offer “instant” sessions. Some 
products and services have both possibilities. The investigation team should determine which session type most fits 
the needs of the organization. The team should also find out if one of the options can be disabled if both options are 
offered. For example, an organization may only want the ability for sessions to be set up in advance and not 
instantly. 

There are many different options for meeting entry and meeting security, depending on the product or service 
that is used. 

• The most basic type of session entry is one that allows the attendees to simply go to a URL and 
automatically become part of the session. 

• Another type allows the attendees to go to a URL, but they must obtain permission from the session leader 
to enter the session area. If the session leader does not grant permission, the attendee will not be allowed to 
enter. 

• Still another type has the attendee go to a URL and then enter a generic session ID and/or password to enter 
the session. 

• The most complex session entry type requires the scheduler or session leader to set up the meeting and 
make a specific attendee list, usually by email address and password. In this type of session, attendees must 
provide their email address and their specific password to be allowed entry to the session. 

Some products allow the session entry type to be flexible with different options, depending on the need for 
security, while others have only one option. The investigation team should find out what options are available and if 
the ability exists to disable one or more of the options. Depending on the needs of the organization, it may also be 
necessary to have even more security for sessions. If this is the case, find out if additional security measures can be 
implemented, such as IP address verification, access control lists, or other measures. 

Facility, Funding, and Usage Issues 

Sometimes, a Webconferencing product or service will create the need for the physical locations to be 
adapted in some way. This may include installing network cabling, phone lines, or other technology connections. 
This could also include the installation of seating, cameras, or more computers, depending on how the product will 
be used. If your organization has not already decided on a physical location where the Webconferencing product 
will be used, such as a computer lab, distance education, or conference room, a determination should be made before 
the final purchase. Some products or services can be used from almost any computer location connected to the 
Internet. Other products contain proprietary hardware that would usually require the session leader to report to a 
location where that hardware is stationed. 
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EVALUATION CHECKLIST 

An Evaluation Checklist will give your team an ordered procedure to follow and ensure that things have not 
been overlooked in the investigation process. Depending on your organizational needs, or the size of your 
investigation team, this checklist may be expanded to go into greater detail for each individual step. It is 
highly recommended that there be an agreement about the steps in the checklist following the formation of 
the Investigation Team. If there are any problems or questions, they should be addressed before the team 
begins the questioning and testing section. 



Needs Analysis 


1) 


1) A thorough needs analysis been completed. 


2) The results of the needs analysis have been interpreted to give a basic idea of what is needed 
from a webconferencing product or service. 


2) 


Formation of the Investigation Team 


3) An Investigation Team has been formed 


m 


4) The Investigation Team represent clients, faculty members, trainers, facilitators, schedulers, tech- 
nical and support staff, administrators, and any other groups that will be affected by the final decision. 


■ 


Questioning and Testing 




5) The team compiles a pool of possible products and services, making every effort to include all 
possible products and services for the initial phase of the investigation. 


■ 


6) The team begins the investigation of individual products. 


6 


a) The team tries to determine if the companies behind the product or service are financially 
stable. This is done by searching the web, business and trade journals and stock tracking, if 
possible. Any companies that are determined to be unstable are eliminated from the pool. 


a) 


b) The team investigates products and eliminates those that do not offer features that were found 
to be absolutely essential by the needs analysis. 


b) 


c) The team attends product demos and uses free trials when possible to help determine which 
products or services are most likely to serve or not serve the needs of the organization as 
determined by the needs analysis. 


c) 


d) Through the elimination process, attendance of demos, and use of free trials the field is 
narrowed to a small number of strong contenders that are most likely to meet all of the needs 
of the organization. 


■ 


Questioning of Current Users 




7 ) The Investigation Team or their agent contacts current users to determine if the product or 
service performs as well as expected and if there were any unexpected problems. Any 
contenders that have been continually problematic for other users are eliminated from the pool. 


7) 


Final Decision on Product 


8) 


8) The Investigation Team discusses the positives and negatives of the final contenders and votes 
to decide the final product. 


9) The Investigation Team submits a request for purchase. 


m 


Purchase and Implementation 




10) The organization or its purchasing agent negotiates a purchase agreement with the product or 
service provider. 




11) The product or service is implemented within the organization. 





Once the decision has been made that a Webconferencing product or service will eventually be purchased, 
a course of action for securing funding must be implemented. Implementing an organization-wide Webconferencing 
product or service can have a very high startup cost. Cost estimates should include the costs of buying any servers, 
hardware, and software that may be required. In addition, estimates should also include the cost of any other services 
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that may be required, such as an audio bridge, full-time scheduler, trainer, and the costs associated with equipping a 
special room for the use of Webconferencing, if necessary. 

If your organization requires one, you should begin work on a purchase request or other similar document 
and paperwork as soon as possible. Some of this work may even be done during the investigation phase so that the 
purchase and implementation process will happen more rapidly. 

When you have completed your needs analysis and your investigation, you should have a comprehensive 
idea of which products are most likely to serve the needs of your organization. When you have your choices 
narrowed down to a handful of possibilities, you should ask the software company or service provider for a list of 
their clients that you may contact to see how pleased they are with the product or service. 

Here are questions that you will want to ask when contacting former clients: 

• Did the product or service perform in the manner described by the sales representative? 

• Was implementation more difficult than expected? 

• Did this product meet the needs of your organization? 

• Were there additional charges that were not anticipated? 

• How difficult was it to obtain product support? 

• Would you recommend this product or service to someone else? 

• Since implementing this product or service, have you changed to or added an additional service of this type 

to your organization? 

While it will not affect your final decision on what product or service to purchase, you should establish policies 
on etiquette, proper use, and scheduling prior to the implementation of Webconferencing. These policies must be 
provided to all users so that there is an understanding of what is expected of them. A list of consequences should 
also be included with the policies so that there can be an expectation of uniform enforcement of the policies. If 
sessions for one group of users are of a higher priority than sessions for a different group of users, this should be 
noted. There should also be a clear policy on which groups or departments within the organization take priority over 
other groups or departments, and why, when there is a conflict in scheduling. 

Conclusion 

With so many different products and service providers on the market, it can be somewhat difficult to make 
a decision on what will work best for your organization. Different features, pricing structures and implementation 
issues should be weighed carefully against one another to find a solution that will work best for your organization. 
The money saved with a lower initial price may be lost if the implementation is costly. Long range costs should also 
be taken into careful consideration if they can be foreseen. The need for continual server and hardware upgrades, 
equipping facilities with special equipment, and the possible need for long-term trainers, schedulers, and technical 
support staff might change the decision from a product or service with a lower initial cost to one with a higher initial 
cost that eliminates or significantly reduces the long range costs. A committed team armed with a thorough needs 
analysis will be able to work together to determine the positives and negatives of each possible avenue and find a 
Webconferencing solution that will most closely match the needs of your organization. In the end, the hard work 
that is put into investigating and finding the proper Webconferencing solution for your organization will pay for 
itself in reduced travel costs, increased productivity, and greater ease of communication and collaboration. 
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This paper describes a pilot program called the Art and Technology Writing Workshop (ATWW). 
ATWW was based on a theory of arts integration and new literacies studies. ATWW supported youth literacy 
development in three ways: writing workshop, theatre arts activities, and technology experiences. The 
participants used writing to mediate between experiences with creative drama and role-playing and the use of 
the computer to create games. The goal was to learn how best to integrate these new literacies in an after school 
setting. The paper includes a description and analysis of the ATWW program described as an activity setting. 
The outcome of this collaborative, formative study has shed some light on how technological story telling tools 
based on games address new literacies with possible implications for other learning communities. 

Introduction 

The paper discusses the first iteration of the Art and Technology Writing Workshop (ATWW) in a 
drop in after school program. We combined writing workshop techniques with theatre arts and computer game- 
making technology to stimulate young people to write. Using a formative study approach (Jacob, 1992), or 
design experiment (Brown, 1992), we looked at how a computer-mediated instructional intervention (ATWW) 
could best be implemented to achieve a valued pedagogical goal in writing instruction (increasing the amount 
and diversity of students' independent writing). The purpose of this preliminary study was to anticipate how best 
to design the integrated curriculum. 

Video gaming is a shared experience among many youth in the U.S. As players they learn a great deal 
about a bounded virtual world in order to play (Gee, 2003; Fromme, 2003). However, it is only rarely part of the 
discourse community in their schools, although there are recent efforts to establish its place in education. In 
fact, many in education are unaware of the impact that new literacies, new media and new ICT (Information and 
Communications Technology) are having on our culture as a whole. (Robertson, 2005). What students 
encounter are contextual, integrated, new literacies in the “post-typographical world ” (Reinking, 1998). The 
term “new literacies” (Leu, Kinzer, Corio and Cammack, 2004) refers to the situatedness of literacy and its 
many creative forms. New literacies studies (Kress, 2003) build on multimodal communication skills and 
interests that students possess. 

Art and Technology Writing Workshop 

ATWW was a collaboration of three entities: a neighborhood community center, a non-profit writing 
program and this university researcher. The faith-based multipurpose neighborhood center provided programs 
promoting leadership development, self-sufficiency, educational support, and gang prevention programs for 
youth. The non-profit creative writing program has been working with Native American students in public 
schools. The author had been working with drama and writing in middle school, and at using electronic game 
engines to tell stories. We were looking at ways to combine literacy and the arts while integrating technology 
into the learning activity system. And, we were hoping to learn what aspects would work best in a writing 
classroom. The challenge is to capture the learning and creativity and focus it on valued literacy skills for the 
classroom. 

Arts Integration Theory 

Arts Integration Theory of education (AIT) is based on the idea that art underlies all the higher human 
functions that involve problem-solving, creativity, and invention and connects those activities to the emotional 
level that is so important developmentally. (Betts, 1999) Art and literacy are both productive social activities 
that are constructive at their core. (Vygotsky, 1978) AIT acknowledges that the child perceives information in 
an emotional context through an affective filter. The aesthetic response mediates between the learner and the 
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environment. (Vygotsky, 1971) and creates a reciprocal engagement of the teacher and pupils, each having an 
effect on the other as art skills and confidence grow in an environment of proximal development. In this case, 
drama activities bring together core curriculum matter and socio-historical development. (O'Day, 2001) In an 
AIT learning environment, children construct their knowledge, process and organize the information and re- 
present in their own terms. (Betts, 2004) Ideally, ATWW would provide a framework for this activity. 

Cultural Historical Activity Theory 

Cultural Historical Activity Theory (CHAT) takes human activity as the basic unit of analysis. CHAT 
views human consciousness as emerging from object-oriented activity (Engstrom 2004) and emphasizes the 
socio-cultural aspects of learning as well as the cognitive, acknowledging the nature of the developing learner. 
CHAT makes it possible to investigate the mediational processes that innovative tools can bring to educational 
settings and takes into account the goals and motives of the learners in the context of the setting and its social 
aspects, focusing on interactivity of the various aspects. 

From the point of view of the youth as subjects in this ATWW activity system, they had as an object 
their writing. They were given several new, mediating tools to help them become better writers. Those new 
tools included word processing software and the SCC program, as well as creative writing exercises and drama 
activities. Reading their work aloud and editing using the remix process they invented were new writing tools 
as well. The outcome of the 10-week workshop was a body of written products, presentation experience, and 
perhaps a new view of themselves as writers. Looking at its culture, the division of labor, or roles available, 
and the rules that were established can describe the context in which ATWW took place. 

Formative Study 

Formative study and design experiments have been increasingly used show to be useful in bridging the 
gap between theory and practice. (Brown, 1992;Collins, 1990) In formative experiments researchers work 
closely with teachers to design, develop, implement, and evaluate classroom innovations. Such activities lend 
themselves to creative processes in learning and in software design. (The Design-Based Research Collective, 
2003) 

Research Questions 

(1) What did the activity system look like? 

(2) What factors in the activity system enhanced or inhibited ATWW's effectiveness in enhancing 

students' literacies in writing and technology? 

(3) How can the ATWW program be more effective? 

(4) What unanticipated positive or negative effects did ATWW produce? 

(5) Did the activity system change as a result of ATWW? 



Setting 

ATWW took place at a long established neighborhood community center. Many of its programs are 
bilingual in Spanish. Its several small buildings housed a meeting space with a kitchen, a newly outfitted 
computer lab, several classrooms, a space for ballet folklorico, and a large yard for recreation. It is located near 
two traditional Native American communities. 

The computer room was in a separate building, a converted house with a kitchen and three rooms that 
had been used as a clubhouse for youth. The interior was painted with solid colors and sealed windows. The 
eight refurbished computers were in the largest room, set up in a circle in the middle facing out under a bare 
light bulb. There was a printer but it didn’t connect to the PC’s. Each PC had word processing and Internet 
connections and we installed StageCast Creator2 ™ (SCC) for the workshop. 

Computer Games 

SCC is an affordable cross platform application for creating two-dimensional computer games. The 
interface is relatively simple and permits students to learn how to position and control the game characters and 
settings. Students are able to “look behind the curtain” at how the games are made and begin to gain control by 
programming the game space. A goal of ATWW was to encourage students to change their stance toward computer 
games from participant in a previously programmed context - as the player, to that of an actor or director controlling 
the movement of the characters and the camera. 
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Methods 



This formative, naturalistic, study relied on combining participant observation and interviewing with 
pre- and post- surveys of perceived self-efficacy and attitude, computer artifacts and writing samples. Analysis 
of student writing was to be based on a rubric used in local schools to determine the new literacies present and 
the best practices in effecting literacy improvement. The University of Arizona’s Institutional Review Board 
approved the informed participant consent protocol. However, the high turn over among the participants gave us 
little useful quantitative data, and only impressions of their writing process. Only two consent forms were 
returned. Only these two students are quoted. Of the seven students who were there the first night, and filled out 
the pre-questionnaire, only one was at the last session to fill out the post-questionnaire. 

Findings 

What we found to describe about this activity system is here organized according to the Activity nodes: 
subject, object, tool, culture, rules and roles. It was clear that the inconsistent attendance made it impossible for 
the program to really get off the ground. ATWW would work better with more consideration of student interest 
and motivation, perhaps by directly addressing reasons that a student would want to come regularly. Although, 
to be fair, these students had no idea what to expect and the program was quite different what had been offered 
before. Some students showed that they were more interested in writing when they got to recognize themselves 
in it and when they got to see their words acted out. This was a hoped for outcome. That the ATWW would be 
such a good inauguration of the computer lab and cause the administration to make adjustments so that it 
worked better was an unintended outcome. For these youth, in this context, each of the three integrated 
activities was novel. Their acceptance of new possibilities and new identities produced a change in the ATWW 
activity system. 

Subject 

The survey forms everybody filled out at the first meeting indicated that this group of nine individuals 
(including staff) had some shared characteristics. They liked hip-hop music (with one exception), they liked to 
write letters to family and friends, and they liked to make up stories. They were not the most sophisticated 
Internet users or cultural mavens. They mostly did not read Manga books or use the Internet for shopping. Most 
said they liked to draw and to act. Most of the youth, but not the instructors (or myself), said they played video 
games a lot. About half the group had computers at home for homework. About half seemed to be invested in 
school work; liking teacher praise for their writing, and writing a lot in school. And the group was split on 
whether a computer helped them write. 

Object 

The object of this activity system was writing. We took a broad view of writing to encompass 
multimodality in narrative construction and creativity. The youth that came were interested in writing. They 
acknowledged at the outset that writing was important for them in school. For the most part during ATWW they 
were productive in their exercises and assignments. They did automatic writing and learned how to rough out 
ideas for later reworking. They worked on descriptions based on the five senses and wrote about their home 
neighborhoods. They learned to act out stories they had written to in order to edit and create dialogue. All these 
activities gave them new ways to think about writing and about themselves as writers. 

Tools 

Students were introduced to a computer game engine called Stage Cast Creator 2 (SCC) to create 
simple two-dimensional computer games and multimedia presentations. Students can create a story and 
visualize it by constructing a graphic narrative including character, goals and obstacles. This relates directly to 
the creative writing process. SCC also allowed students to import graphics and create text to report research on 
a topic using multimedia. 

Students were introduced to automatic writing exercises and learned from the experience the important of 
editing and revision. For example, they wrote about “Where I’m from...” Some wrote about their neighborhoods, 
where “sirons [sic] are like music.” A young lady from southern California wrote about missing her gang and her 
“homies.” A boy from Mexico remembered music and the “smell of burnt wires.” 

Students learned several drama games like “Show an Emotion” game where they had to mime an emotion 
in order to sit down. The youth were wary of being called on to perform in company. As the more extroverted lead 
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the way, the boys and girls became more relaxed the group participated with ideas and enthusiasm. “They 
encouraged each other to try,” wrote the drama teaching artist. 

Culture 

Each of the weekly meetings of ATWW had a different make up as youth dropped out and new ones 
came to participate. The culture that was formed was loose and largely shaped by the staff with the different 
individuals who came. There was intensity to the workshop style that engaged most of the youth. They looked 
for ways that it was like school and for ways that it was like hanging out and seemed to generally decide that it 
was neither. 

The context was friendly and supportive of students. Within this open atmosphere, the 8 or 9 students who 
participated off and on in ATWW were fairly engaged while there. They were productive, and able to revise their 
own work and constructively criticize the work of others. 

There was a drop-in atmosphere that did not encourage consistent attendance at ATWW. There were issues of 
behavior and mutual respect in the early weeks that required explicit rule making by all concerned. 

Rules 

The rules were adapted from the unstated rules that were in effect during the previous after school 
Prevention Program. It was established that the person who was reading would get the attention of the group, 
that there would be no cell phone use and that people would stay on task until the breaks. During the later weeks 
they were negotiating rules for themselves. 

Roles 

Students explored various roles as writers, editors and critics of their own, their peers and other writers’ 
work. The age range was greater than we had estimated and there was a good deal of cross-age interaction, as 
sometimes the younger knew more about using the computer than the older youth and could assume the role of 
teacher. They coached each other in reading aloud and collaborated freely in creating dialogue to match each 
other’s story ideas. Writing in an out-of-school context seemed to allow them to approach expressing their ideas 
more freely. 

Outcomes 

It was a very productive workshop for the youth who came. They were enthusiastic writers, ready to listen to 
themselves and to others, and willing to revise and refine their work based on the theatre activities and the game- 
making goal. Since the data collection was hampered, a simple review of the Traits (Ideas, Organization, Voice, 
Word Choice, Fluency, and Conventions, plus Presentation) (NWREL, 2001) showed that they were learning to use 
language in more creative ways. 

Obviously, more research is needed in order to create a syllabus that will incorporate these three elements. The 
ATWW will be implemented in other contexts, and with more control and attention to student motivation, we hope 
to learn more about how art and technology integration works in writing instruction. 
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Around the World - Crossing International Boundaries 

Suchita Bhatt, Montclair State University 
Matthew Conforth, Passaic Valley High School 



Description 



The “Around the World” project is a program that strives to promote global awareness and understanding through a 
combination of videoconferencing, on-line learning projects and discussions, E-mail and other classroom 
technologies. The main objective of this project is to have the world audience become effective and responsible 
contributors to problem-solving at the individual, community, national and international levels. The presentation 
demonstrates the use of various communication technologies in bringing the world together. 

Abstract 

1. Issue 

At Passaic Valley High School (PVHS) a need was felt to design an international awareness program that reflects 
both New Jersey’s Core Curriculum Content Standards and the Cross-Content workplace readiness standards. A 
program was to be created that met our goals of (1) enhancing the awareness, understanding and tolerance among 
participants for different value systems and cultures (2) encouraging and fostering student-to-student exchange and 
increasing opportunities for both students and teachers to experience activities focused on global awareness (3) 
creating effective and responsible contributors to problem solving and decision making processes at individual, 
community, national and international levels (4) aiding participants in exhibiting reflective attitudes toward their 
personal values, the values of others, and the cultural values evident in their own and other societies (5) bringing 
youth face-to-face to meet and to discuss world issues that affect and unite us all (7) developing and applying skills 
related to acquiring, organizing, evaluating and using information technology as a communication tool 

2. Description of the project 
Historical Background 

Colin L. Powell, in a speech on International Education Week 2002 stated, “As we work to end the scourge of 
terrorism, let us also work to increase peace, prosperity and democracy. We can do this through international 
programs that promote the exchange of ideas and the sharing of experiences. These programs give us insight into 
other languages and cultures and in the process build long lasting relationships among people based on mutual 
understanding, respect and trust.” 

The “Around the World” project has been ongoing for the past two years. Along the way lessons have been learned 
and changed innovated. 

People Involved 

The “Around the World” project represents a unique partnership between K12, higher education, and business. The 
major contributors to the project are Passaic Valley High School, the Verizon Access New Jersey Portal, 
NJEDge.net , Montclair State University and Global Education Motivators. 

Solution 

The “Around the World” project brings together teachers and students from many different countries in 
collaborative projects that help us better understand each other and the world in which we live. Students and 
teachers exchange ideas and information on topics of interest through videoconferencing and an online community. 
The highlight of the project is the 24-hour, around the clock and around the world series of videoconferences. 
Students from PVHS stay in school overnight to accomplish this task. This allows for students in schools in all 
different time zones around the world to meet with PVHS students during the normal course of their day. PVHS 
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prepares the students for videoconferences by assigning student “ambassadors” for each participating country. Each 
ambassador is required to research their assigned country in terms of its history, culture and current events. They 
then present this information to their classmates. Ambassadors also moderate and facilitate the videoconference 
with their assigned country. 

Videoconferencing with each country is for an hour. Schools prepare some form of presentation for videoconference 
that will help other leam more about them. Students are exposed to creating videos and using Power Point as 
presentation tool. Participants also prepare at least five questions to ask about the other school. These video 
conferences also cover a wide range of topics depending on the requests of the teachers and students in each 
country. 

Between the videoconferences, students engage in on-line discussion on the “Around the World” online community. 
Instructional design support and training are provided to our teachers for creating the online community. Teachers 
from each participating country are enrolled as leaders in that community and the students as participants. Teachers 
from each country post various documents and related web sites on this community. Teachers are also responsible 
for starting a discussion forum and mentoring it. Students can post information, make comments, and continue 
discussions for several weeks before and after the 24-hour “Around the World” videoconferences. 

In addition, each videoconference is streamed on the Internet. This permits parents, community members, and 
students from other schools in the USA and around the world to view each event as it happens. The 
videoconferences are also archived for future educational use. 

3. Outcome 

The overall assessments of each year’s projects indicate several important changes in student knowledge, attitudes 
and performance. The majority of students involved in the projects learn that the world has a much different view of 
them than they had anticipated. One student stated, “I learned that we (Americans) don’t know nearly as much as we 
should about the rest of the world”. A majority of them also demonstrated a change in their perspective on global 
issues and a better understanding of cross-cultures. One student commented, “I learned that most kids around the 
world are just like us”. Students also tended to be more motivated to acquire knowledge about the country and 
culture of the students they were collaborating with in a project. An excellent example of this was demonstrated 
during our first 24 hour “Around the World” project. The first videoconference of the project clearly indicated that 
the students from other countries knew more about technology tools, came up with better presentations and in 
general they had a better knowledge about our culture. This fact motivated our students to use time between 
videoconferences to come up with better presentations, research more about the technologies available to them and 
find out better facts about the next country on the schedule. 

4. Relevance 

The “Around the World” project could be seen beneficial to K12 educators, NJEdge.net Constituent Services 
Representatives, NGOs and international schools and universities. This teaching and learning project encourages the 
integration of different Internet technologies and promotes interactivity, student collaboration, reflection and 
encourages a learning environment that does not end when the class session finishes. 

This presentation will demonstrate to other institutions how we, as educators, can become learning mentors in a 
process that will require students to accept more responsibility for learning and to be more active partners in the 
process. The presentation will demonstrate how a local school can strategically engage in distance learning while 
partnering with outside resources and helping our youth become global citizens. 

Conclusions 

It is important to remember that the United States is such a dominant force in the world that most of our students 
tend to lose sight of how we appear in the eyes of the global population. News and other media also provide a 
distinct bias in how we see ourselves and how we are portrayed. If we are ever going to change that view, it must be 
through the education of our youth. Our students need to learn not just what is happening in the world, but why. 
More importantly this must be understood from a global, cross-cultural perspective. Our students must also gain 
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experience in how to work cooperatively with citizens of other nations to solve world problems and to achieve world 
peace. Through these international projects our students are given that opportunity to truly examine who we are 
from a global perspective. As the students interact they gain an awareness and understanding of their counterpart’s 
perspective on the issues. They are also able to further share and debate their ideas and views as they seek to find 
solutions to global problems. Students are then challenged to reexamine their views and derive new conclusions. 
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Processes for the Design of Web-based Anchored Lessons 

Carol Brown 

Introduction 

Classroom teachers will use technology when there are assurances that the lessons are effective for 
achieving instructional goals, not plagued by multiple obstacles in the use of the technology, and when K12 students 
are motivated to complete the lesson assignments. The simple use of embedded hyperlinks within a word processed 
document can be used to design and develop anchored instruction. All that’s needed is word processing software, 
connections to the Internet, and ideas for a community based problem scenario. Because anchored instruction 
originally used high interest stories with video technology, this study included multimodality anchors using links to 
a variety of online resources providing streaming video. Most school systems in North Carolina provide 
subscriptions to online video resources, thus motivation was high, for graduate students to design and develop video 
anchored lessons for their students in the classroom. 

Review of the Literature 

Anchored instruction is a category within constructivist learning environments and closely related to 
situated learning theory. Typically a high-interest story has imbedded anchors with information needed to solve a 
complex, multifaceted problem depicted in the story. Subject matter is communicated through anchored resources 
within a realistic story rather than lecture or textbook readings. According to Bransford (1992) and his colleagues at 
the Cognition and Technology Group at Vanderbilt (CTGV), anchored instruction provides an authentic setting in 
which students can think as expert problem solvers. Concepts and principles related to a topic are used to generate 
solutions rather than being stored as “inert knowledge” (p. 293). Anchored instruction was originally designed as a 
means to guide students in making the best use of available information. Students learn to see the relevance of 
information as needed for solving a particular problem (Bransford, et ah, 1989) Branford and the CTGV (1990) 
reported the use of video-based lessons using an authentic story that would be motivational for students. The videos 
were used very differently than typically used in K12 classrooms. Rather than simply presenting a documentary 
film, CTGV videos were used to provide information within a rich storyline that would motivate students to solve 
problems. The motivation factor is important and facilitates the problem-solving process (Ormrod, 1999). Other 
studies in motivational problem solving defined the anchored lesson as one supporting a complex problem that 
students will consider “worth solving and that validates the learning of a set of relevant skills and concepts” (Barab 
& Landa, 1997, p53). Research suggests that highly motivated students are more likely to exhibit intrinsic 
motivation when a topic is of personal interest. When an activity is meaningful, a student is more likely to set his or 
her own learning goals, process information more efficiently, and engage in the activity “over the long haul” 
(Ormrod, p. 441). Students transfer knowledge best when the original learning occurs within contextualized learning 
experiences thus the importance of authenticity for both the storyline and the embedded data. Anchors with the 
imbedded data support several steps within the problem solving process and likely lead to reflection and 
metacognitive thinking (Baker & Wedman, 2000). 

Based on earlier work of Bransford and the CTGV, Love (2005) designed video-based lessons that support 
instructional methods for preservice teachers. Case studies of elementary classroom language arts instruction is 
viewed by preservice teachers who use the option to review the videos as many times as needed to identify best 
practices for teaching language and literacy. Using complex problems situated within a small problem space, 

Love’s anchored lessons were designed to overcome the problem with “inert knowledge” by providing cognitive 
apprenticeships for preservice teachers. A cognitive apprenticeship provides the opportunity for a novice (preservice 
teacher) to observe an expert (inservice teacher) solve problems and exhibit best practices in a real classroom 
setting. According to Love: 

“A vital feature of anchored instruction is the use of a macrocontext or ‘complex problem space’. . 

. . The importance of using realistic macrocontext is paramount to the success of the problem 
solving. Like cognitive apprenticeships, the use of realistic, interesting macrocontext permits 
students and teachers to experience many of the problems, situations, tools and environments from 
real-world situations (p.302)”. 
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In addition to cognitive apprenticeships, the multimodal options available through current technologies provide 
greater options than ‘‘‘Jasper’ (Jasper Woodbury, available online) ever imagined. Through the use of online 
resources, meaning is generated in multimodal format of audio, video, hypermedia, images, and any combination of 
all. Unlike Jasper, the teacher (or designer) no longer is required to shoot original video, but has the option of 
multiple Internet resources from which to anchor the problem. Love identified seven important design elements 
within multimodal anchored instruction. These are: selection of appropriate anchors; shared expertise and discussion 
on the problem statement; expanded anchors to provide diverse perspective; transfer concepts from the anchors to 
different context; problem with anchors tied to instructional goals; additional anchors using a variety of formats; and 
opportunities to explore the anchors (p.303). 

Current use of video for anchored instruction includes streaming video as well as downloaded clips through 
online resources such as “United Streaming” (www.unitedstreaming.com). Even though a subscription website, 
many school systems in the region of our university are finding the streaming videos useful for classroom 
instruction. Teachers may show the entire contents of the video or a small, pre-selected, clip that contains specific 
information needed to solve the problem within the anchored lesson. Similar to codes used with videodisc 
technology, the design of this resource, permits bookmarking of specific video clips timed from one to several 
minutes in length. 

The potential for gaining problem solving skills that will stay with the student beyond the school setting, 
opportunity to engage in meaningful community based experiences, and reflective dialog are great with the use of 
anchored instruction. For this paper, we will report the results of the content analysis for anchored lessons that are 
community-based, with complex embedded data in the form of web links to html resources and/or streaming video. 
The technology is very simple and only requires the use of inserted links in a word processed document; however, 
the motivation factor was high and the quality exceptional for the graduate students who designed the lessons. 

Methods 

The subjects for our study are graduate students enrolled in a master's degree program for instructional 
technology. One required course assignment for their degree program includes the design and development of an age 
appropriate anchored lesson module that supports the state’s standard curriculum for K12 schools. The design of the 
lesson must include the use of embedded data in the form of web links to a variety of resources related to a case 
study within their communities or nearby region of the state. The anchored lessons that were analyzed for this study 
are unique in that they are developed by teachers from a wide geographic area across the state of North Carolina. All 
are distance education students enrolled in an Internet delivered course, thus their case studies and situated problems 
reflect diverse perspectives from across the state. Topics and problem scenarios that might be of interest in the rural 
mountain region of the state may vary greatly from interests in urban areas in the central part of the state. The 
anchored lessons were archived over four semesters. 

Data for this study span four semesters and two years time- spring 2005, spring 2006, spring 2007, and fall 
2007. Three of the four groups were enrolled independently as graduate students in the Master of Education in 
Instructional Technology. The students are licensed teachers, media or technology specialists working in varying 
grade levels and content areas. The fourth group consists of a cohort of 15 students, seeking the Master of Education 
in Instructional Technology. The cohort consisted entirely of classroom teachers and a technology facilitator. Cohort 
members are on a predetermined timeline and sequence of courses. They also have the advantage of living and 
working in close proximity, thus providing more peer support than in other online degree seeking students. There 
were no media specialists in the cohort group. In Table 1, see a summary of group characteristics by time, 
professional background, and variations in the design of the course in which they were enrolled. 
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Table 1 . Group characteristics and summary of mean scores from evaluation of lessons. 



Group 


Eval of 
lesson 


N 


cohort 

y/n 


Peer eval 
(feedback) 


Instructor 

feedback 


Reflections 


Pre- 

Post 


Geographical 

region 


Spring 

05 


M=1.8 


15 


No 


Yes 


No 


No 


No 


Across the state 


Spring 

06 


M=1.6 


19 


No 


Yes 


No 


No 


No 


Across the state 


Spring 

07 


M=1.7 


13 


No 


Yes 


No 


No 


No 


Across the state 


Fall 07 


M=1.8 


15 


Yes 


No 


Yes 


Yes 


Yes 


Piedmont area of 
the state 



The student-generated anchored lesson was assigned in one unit of study while students were enrolled in one of 
four semesters Spring 2005, spring 2006, spring 2007, and fall 2007. An online lecture written by the course 
instructor was included in the course materials. Readings from the research literature on anchored instruction, 
situated learning, and a review of the “Jasper Woodbury Series ” provided the base knowledge needed for the 
graduate students to develop the assignment for the anchored lesson. Criteria for the lesson included: 

• An authentic problem scenario related to the community in which you and your students live 

• Embedded links to electronic resources in the form of teacher produced documents, online web resources 
or streaming video pertinent to the problem scenario 

• Objective(s) for the lesson that are correlated with the North Carolina Standard Course of Study 

• Minimum 200-250 words in the problem scenario 

Groups 1, 2, and 3 were instructed to submit their anchored lesson to an online discussion forum and participate in a 
peer review session through chat and discussion threads. Group 4 were instructed to deliver their anchored lesson to 
a group of K12 students in a nearby community; administer pre and post tests to K12 students receiving the 
anchored lesson, and write a personal reflection at the conclusion of the project. Groups 1, 2, and 3 submitted the 
final version of the anchored lesson following peer evaluations and small group discussion. 

Based on the review of the literature, six main elements were identified as important for the design of effective 
anchored instruction. Each of the 45 lessons submitted by graduate students was reviewed for evidence of the 
following: Do the anchored lessons include characteristics that result in motivational problem solving experiences 
for K12 students? Would the problems promote transfer of learning to their community and potentially contribute to 
their productive involvement in the future? Are the anchors age-appropriate, providing the cognitive links that help 
students generate problem solutions that develop skills likely to be used in a new and different problem? Would the 
anchored lessons provide legitimate learning experiences that students would likely consider “worth solving”? 
Based on these four main elements, six criteria were established for the rubric used to evaluate each of the 45 
lessons used for this study. See Table 2 for a list of criteria used to evaluate the design of the anchored lessons. 
Lessons were blind reviewed twice, once by the researcher and once by the instructor. Mean scores for all groups 
are summarized and presented in Table 3 

Table 2. Rubric used to evaluate the design of anchored lessons. 



Student ID 


Yes = 2 
points 


No = 1 
point 




Yes = 2 
points 


No = 1 
point 


Adequate anchor 






Age appropriate 






Problem worth 
solving 






Motivational 






Transfer to 
community 






Complexity of 
problem 






Means 
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Table 3. Summary of evaluation of the design of anchored lessons over a four semester span of time. 



Evaluation of Design Components in Anchored Lessons 




a Adequate Anchor 
B Problem worth solving 
E a Transfer to community 
■ Age appropriate 
n Motivational 
□ Complexity of problem 



Y=2 N=1 M=1 .72 



Analysis of each lesson showed whether (or not) there was evidence of the six elements needed for effective design 
of anchored lessons. An evaluation of 45 lessons from Groups 1,2, 3, and 4 resulted in a mean score of 1.70 for the 
criteria “Adequate Anchor”. The evaluators looked for evidence related to necessary information for problem 
solving, within interesting and age appropriate resources. The criteria for “Problem worth solving” (1.78) were 
evaluated based on design of problems that were suited to the macrocontext used for the community scenario. 
“Transfer to community” (1.85) was an interesting criteria that linked the worthiness of the problem to the diversity 
of community settings. Problems included topics related to lonely elders, heavy backpacks for students, sleep 
deprivation of students, service in soup kitchens, bullying, character education, drunk driving, fundraising, teen 
pregnancy, drought in North Carolina, conflict management, and stray pets. Anchored lessons were evaluated most 
highly for this criterion. The lessons also were designed for age appropriate vocabulary and content of anchors 
(1.80). The lessons received the lowest scores for “Motivational” (1.63) characteristics, and “Complexity of the 
Problem” (1.55). The quality for one fourth of the lessons was evaluated at the level of textbook worksheets with 
dull unimaginative problems. One fourth of the lessons were designed with problems requiring only one step 
solutions or gathering of basic facts for a report. The problems were not representative of a complex problem 
requiring multifaceted steps. Many of the lessons, however, were quite excellent and did match the criteria as 
summarized from the literature. A sample lesson is available in Appendix I. 

Group 4 submitted the final version of their anchored lesson after pre and post tests were administered and 
the lesson delivered to K12 students. In most instances, students worked in small groups and engaged in discussions 
related to the problem statement. A rubric with specific criteria matched to characteristics for anchored instruction 
was used to evaluate the quality of the anchored lesson. Using Love’s model (2004), the rubric used for a second 
evaluation of Group 4 included evidence within the lesson for 1) selection of appropriate anchors, 2) opportunity for 
shared discussion, 3) transferability of the problem, 4) anchors tied to instructional goals, and 5) includes anchors 
using a variety of formats. The results of this analysis are in Table 4. Even though pre and post tests were 
administered to each group of K12 students, not all scores were reported. A copy of the tests was included with each 
lesson submitted by Group 4. 
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Table 4. Summary of evaluation of Group 4 for use of small groups with anchored instruction. 



Evaluation of Lessons for the Effective Use of Anchored 

Instruction. 

Possible scores range from 0 to 1. n=15 



1 0.84 0.84 


0.86 




0.88 




0.89 






























































Shared Tied to Multimodal Appropriate Transferability 

discussion instructional formats anchors 

goals 



As can be seen in Table 4, graduate students applied a high standard of teaching methods with the design of their 
anchored lessons. Many of the lessons included opportunities for group discussions. All students carefully aligned 
the content and problems with instructional goals established by the North Carolina Standard Course of Study. A 
variety of appropriate resources were used and problems were effectively transferred to community settings. In 
particular, the video clips were selected based on authenticity for the problem and age appropriate content. 

Document analyses of personal reflections were conducted. Comments typically appearing in reflections 
submitted by Group 4 are in Table 5. Through their reflections the students were able to think about the outcomes 
from pre and post tests and the effectiveness of their lessons. They were able to assess K12 students’ response to the 
lessons. One common attribute in many of the reflections was the rich dialog students engaged in to discuss the 
problems, solutions, and possible products of learning that could be developed. Students even as young as 1 st grade 
were reported to have contributed to problem solving through class discussions. 

Table 5. Common Themes in Personal Reflections 



Results of Pre Post Tests 


Teacher Reflections 


The results from a post survey suggested, not many first graders knew 
what recycling was but were concerned about helping our earth and 
wanted to do their part. None of the first graders knew that water was 
our most precious natural resource and that they could help save it. 
Most first graders answered that they leave the water running when 
brushing their teeth and left the lights on when leaving a room. After a 
group discussion and viewing of the video students were motivated to 
do their part and together came up with an Action Plan and an 
additional recycling project. After this short unit a post survey was 
given and all of the students knew the definition to recycling and that 
water was an important natural resource. All of the students answered 
that they would do their parts by recycling and reducing. 


I really enjoyed this lesson, I believe it was more enjoyable and 
meaningful for the students because they had a part in it. Once 
introducing the problem they were motivated to come up with 
solutions on their own. They left feeling as though they could make a 
difference and I could tell this was important to them. The anchors in 
the lesson helped to give additional resources for the students to learn 
about the topic of discussion. 


The students were really tuned in to the problem’s presentation. They 
didn’t realize that low test scores were such a problem. Several 
commented that their parents had discussed the low scores when they 
read an article in the local newspaper. Most of the students also 
admitted to never thinking much about problems that Spanish speaking 
students might have, until they completed the survey. These honor 
students discussed the fact that the only subjects shared with Spanish 
speaking students were electives. Some truly believed that a Spanish 
version of the EOC tests existed. Fifteen out of the nineteen honors 
students had taken or were currently enrolled in a Spanish class. This 
led the group into a discussion about the difficulty they had in the 
beginning of Spanish class when the Spanish teacher spoke nothing but 
Spanish for an entire class period. 


The students completed their pre survey and I was able to share the 
problem scenario with the students along with a class discussion on 
the first day. On day two, I listed their reinforcement activities from 
the pre survey and the discussion from day one led to the addition of 
other activities. The honors students were really engaged in the 
explanation of their activities. We visited the computer lab on day 
three. The students experimented with Powerpoint, recorded their 
voice using the Audacity audio program, and searched United 
Streaming for an appropriate video to match their assigned unit. The 
last day was spent fine tuning their activity ideas and completing 
their post survey. 
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Results and Discussion 

Groups 1, 2, and 3 were assigned peer reviews within small group clusters in an online (Blackboard) 
conferencing environment. Each teacher uploaded his or her anchored lesson to a discussion forum. Members of the 
small group read and evaluated the lesson. A rubric was used to guide the quality of feedback and ensure useful, 
reliable peer evaluation for all those involved in the discussions. Further analysis of the lessons indicated that 
teachers were able to design well-crafted activities with anchors imbedded within the story-problem. In Group 4, 
graduate students received a more in-depth online lesson with instruction in the theoretical foundations related to 
anchored instruction, including the history of the situated learning and the problem of “inert knowledge”. The 
students also participated in online chat sessions with the course instructor to answer questions and clarify the 
meaning for anchored learning. Group 4 received instructor evaluation and opportunity to revise. Groups 1, 2, and 3 
received peer evaluation and opportunity to revise. All groups received feedback using rubrics designed to evaluate 
the design of the anchored lesson. 

Two main advantages were identified within the initial review of projects: For Groups 1, 2, and 3 teachers 
and specialists worked collaboratively to analyze, evaluate, and propose revisions for the final versions of their 
anchored lessons, and secondly, problems related to nearby community situations were highly motivational for the 
teachers. The online lesson included links to resources with information related to community based authentic 
problems as instructional method. (Kansas Association for Teachers of Science, 2004). We predicted this 
enthusiasm would transfer to K12 students in the classroom. Analysis of documents from Group 4 suggests that K12 
students would be motivated to complete the lesson because of authenticity of the problem. Only group 4 
administered pre and post tests. The purpose in adding this requirement was to gather additional data that might 
suggest that anchored lessons were effective for increasing student achievement. Results from the pre and post tests 
were reported in the reflective writings required for Group 4. For each of fifteen lessons submitted for fall 2007, 
teachers reported an increase in means scores between pre and post tests administered to students. A variety of tests 
were used, including short answer or multiple choice, essay questions, and opinion surveys. For content areas 
requiring recall for End of Grade tests, teachers preferred the objective paper pencil tests. For lessons related to 
subject such as Flealthful Living, Social Studies, or Economics, teachers used opinion surveys designed to show a 
change in students’ adoption of a particular habit or attitude related to healthy lifestyle. In addition to the tests, the 
anchored lessons included some type of student product such as poster, position paper, or slide presentation. This 
provided opportunity for peer and/or teacher feedback and dialog to discuss the problem prior to taking the post test 
to assess achievement of instructional goals. 

A careful analysis of lessons over the four semesters shows that graduate students were able to develop 
lessons that include problems within a community based context and that the problems were designed to transfer to a 
different context for probable expanded learning by the K12 students. The design of the lessons, including problems 
set within a macrocontext were designed to be authentic and related to students’ real-world experiences. Rubric 
criteria for “motivational” and “complexity of problem” received the lowest scores when evaluated by course 
instructors and researchers. Those rated low in motivation were designed as worksheets with problem statements 
similar to those found in classroom textbooks. These were not related to the local community nor set within context 
typically considered “real-world” for K12 students. The second area receiving a low score was in the design of the 
problem statement. Problem scenarios requiring one step solutions, fact based products, and simple recall for pre and 
post tests were evaluated at a lower score using the evaluation rubric. 

Over a four semester span of time, graduate students in school library media and classroom teachers have 
submitted anchored lessons using simple word processed documents with embedded links to multimodal anchors 
that were carefully aligned with the state curriculum. Personal reflections submitted by the graduate students suggest 
that the lessons were interesting in design and a motivational experience during the development and delivery of the 
lessons. Students in the K12 classrooms responded favorably to the lessons. Evidence for this is suggested by two 
themes emerging from the personal reflections submitted by graduate students: First, designing and delivering the 
lessons was enjoyable. They also are motivated to use resources that provide specific data needed to solve problems, 
thus avoiding aimless wandering on the WWW. Secondly, K12 students were motivated to view the anchored 
videos. Each of the 15 graduate students in Group 4 reported that anchored instruction had a positive effect on their 
students’ mastery of concepts or positive change in behaviors related to character attitudes or healthy lifestyle habits. 
Only one graduate student (classroom teacher) reported technical problems in the development of the anchors. The 
use of simple technology greatly increased the focus from obstacles in use of technology to focus on content for 
learning. 

This report adds to the evidence already established that anchored instruction facilitates problem solving, 
transfers from macrocontexts to a different environment, is motivational for both teacher and students, and is best 
used when students have opportunity for engaged discussion related to an authentic problem. 
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Appendix I. 

Note: Key words within the problem statement included anchors to online resources, including video clips related to 
the problems. 

Anchored Instruction for Problem Solving and Higher Order Thinking 
Problem and Anchored Lesson Narrative: 

Leon County High School has experienced an increase in their student population of limited English speaking 
students. The majority of these students speak Spanish. The Hispanic population makes up 23% of Leon County 
school system’s demographics. Students who have limited English speaking and reading abilities are at a 
disadvantage when it comes to passing required North Carolina EOC tests, especially understanding vocabulary. If 
a student doesn’t know the meanings of the words in the definition, how will they ever comprehend the vocabulary 
word? This presents a problem for the school system when EOC test scores for Leon County High School are below 
the state average. The tests are printed in English and some students do qualify for read-aloud by request test 
modification, but the tests are still read in English. Leon County High School offers sheltered classes for their LEP 
(Limited English Proficiency) students in English, Biology, US History, and Civics and Economics. The instructors 
for the sheltered classes prepare their instruction according to the SIOP (Sheltered Instruction Observational 
Protocol) Model. The sheltered instructors for Leon County High School attended staff development training for the 
SIOP Model through the SIOP Institute. Despite the progress made by the school to improve test scores, the 
problem of low scores still exists. Administration is working on the problem by offering LEP students sheltered 
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classes with trained instructors. The school is continually looking for additional methods of instruction to reach the 
LEP students in specific content areas. Administration wants input from native English speaking students, who have 
experience in passing EOC tests. Your tenth grade honors class, of Civics and Economics, has been assigned the 
task of preparing a reinforcement activity for the “Bill of Rights and Citizenship” unit of your course. This activity 
should help provide assistance to the LEP students who have to take the Civics and Economics EOC. You will 
create a list of brainstormed ideas along with your reasoning and submit them to your instructor. 

Action: 

The tenth grade Civics and Economics honors class (19) was given the following survey before they were presented 
with the problem of low EOC scores at Leon County High School. The guided presentation was given with the use 
of an LCD projector and computer, so the students could experience the use of anchors and scaffolds. The Civics 
and Economics teacher agreed to let me work with her class for four days. We had about six hours together. Their 
teacher chose the “Bill of Rights and Citizenship” unit, because she felt this was a citizenship activity. 

The students were given the same survey following the presentation and then were asked to create their 
brainstormed list of supported ideas. 

Survey Questions: 

Do you think EOC testing is a problem for Hispanic speaking students? If so, explain. 

Do you think Hispanic speaking students get any special treatment during EOC testing? If so, what 
assistance do they receive? 

What methods does the school use to prepare Hispanic speaking students for EOC testing? 

The Hispanic speaking students take the same Civics and Economics EOC as your class does. What 
methods would you use to prepare those students for the Civics and Economics EOC? 

Pre survey responses: 

1. 100% of the responses were yes. The students felt that even if the Hispanic speaking students could 
understand English that they probably couldn’t read it. Many Hispanic speaking students stay out of school 
on testing days. 

2. 73% - yes They get to take the test in a separate room, where they get more time. They get to take a 
test written in Spanish. Someone reads the test to them in Spanish. 

The Hispanic speaking students get to take the test in the computer lab, so the computer can read it to them 
in Spanish. 

3. 27% - no The Hispanic speaking students take the test like everyone else, but fail because they 
don’t understand it. They don’t even know how important it is to pass the test. 

—Have special classes the Hispanic students go to for help. 

—A Spanish speaking teacher helps the Hispanic students, while they stay in the regular class. 

—The teachers let students work in groups to help the Hispanic students review. 

—Nothing is done to prepare the Hispanic students for testing. 

flash cards for vocabulary 

practice tests translated into Spanish 

grouping students - one Hispanic student who speaks English with those who don’t. 

Hispanic students can key English words into an electronic Spanish translator or computer program. 

Post survey responses: 

1. 100% of the responses remained yes. Two additional responses that were added include: Hispanic 
speaking students probably don’t have parents or siblings at home, who can help them with their school 
work and the Civics and Economics class is full of vocabulary. It is probably just too many words for the 
students to learn, especially when they may have to leam words in the definition first. 

2. 90% - yes They get to test in a separate room with extra time and teacher who will read the 
questions in English if the students ask them to. 

3. 10% - no The Hispanic speaking students have to take the test in English. They should get the 
chance to take it in Spanish. If they know the material, it shouldn’t matter what language they show their 
knowledge in. 

—Have special classes the Hispanic students go to for help, where teachers are specially trained to help 
them. 

List of brainstormed ideas: 

A Spanish speaking teacher helps the Hispanic students, while they stay in the regular class. 

The teachers let students work in cooperative learning groups to help the Hispanic students review. 

The students work in cooperative learning groups to create flash cards with vocabulary from the “Bill of Rights and 
Citizenship” unit. The words and definitions are printed in English and Spanish. 
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* A podcast could be created where students take turns reading the vocabulary words from the flash cards and their 
definitions. The podcast could be saved to the student resource folder on the common drive. The instructor can 
reserve the computer lab, so the Hispanic speaking as well as other students can listen to the podcast as they look at 
their flash cards. They can also use the computers in the Media center during lunch and before and after school. 

The instructor could save the podcast to her school wiki and the students, who have computers, could listen at home. 
Students could also save the podcast to MP3 Player or Ipod device. 

* An electronic version of the flash cards could be created using a multimedia program such as Powerpoint. Two 
slides per vocabulary word could be created. One slide for the word (Spanish and English) and one slide for the 
definition. Students could create audio links for the words and the definitions. Images could also be added to 
provide additional hints. An audio link for Spanish and one for English. The definition slides could also provide 
links to words in the definition that Hispanic speaking students may have problems with. Again this multimedia file 
could be saved where students could view it individually. 

* The notes for the “Bill of Rights and Citizenship” unit could be given with the use of multimedia. A combination 
of text (Spanish and English), images, audio (Spanish and English), and video could be used to present the 
information. The video could be added from United Streaming. The video would play when the students clicked on 
it. Questions that correspond to the video could be given to the students prior to viewing it, so they could answer the 
questions as they watch. They can play the video as many times as needed. The instructor may want to partner an 
English speaking student with a Hispanic speaking student for this activity, because the video is in English. The 
English speaking student could stop the video at specifics points to show their partner where the answers match the 
questions. United Streaming has a video called “American History: Foundations of American Government” that 
has a segment called “Establishing the Bill of Rights”. 

* A multimedia program could also be used to create a review. Multiple choice questions could be placed on 
separate Powerpoint slides. Each slide will also contain the 4 answer choices that correspond to the matching 
question. Links to match each answer choice to the correct or incorrect choice would be added. The question and 
answer choices could also have audio links that students could click to hear the questions and each answer choice 
separately. Images to provide extra hints and saving for individual student viewing can be added to this activity. 

Student Response Observation and Personal Reflection: 

The students were really tuned in to the problem’s presentation. They didn’t realize that low test scores 
were such a problem. Several commented that their parents had discussed the low scores when they read an article 
in the local newspaper. Most of the students also admitted to never thinking much about problems that Spanish 
speaking students might have, until they completed the survey. These honor students discussed the fact that the only 
subjects shared with Spanish speaking students were electives. Some truly believed that a Spanish version of the 
EOC tests existed. Fifteen out of the nineteen honors students had taken or were currently enrolled in a Spanish 
class. This led the group into a discussion about the difficulty they had in the beginning of Spanish class when the 
Spanish teacher spoke nothing but Spanish for an entire class period. They compared this experience to what the 
Spanish speaking students were experiencing everyday in all their classes. One honor student even spoke to the fact 
the Spanish speaking students probably couldn’t receive any help with their school work at home. No one in this 
group knew the acronym LEP. The honors group agreed that the sheltered classes and SIOP instruction were good 
things the school was doing to help the Spanish speaking students. They felt like they could come up with 
additional activities to reinforce learning. 

The students completed their pre survey and 1 was able to share the problem scenario with the students 
along with a class discussion on the first day. On day two, I listed their reinforcement activities from the pre survey 
and the discussion from day one led to the addition of other activities. The honors students were really engaged in 
the explanation of their activities. We visited the computer lab on day three. The students experimented with 
Powerpoint, recorded their voice using the Audacity audio program, and searched United Streaming for an 
appropriate video to match their assigned unit. The last day was spent fine tuning their activity ideas and 
completing their post survey. The students wanted to use the brainstormed list to answer number four. Another part 
of their discussion that was important to them was independent study. Most of the honors students felt that any 
student, English or non-English speaking, should have activities that could be practiced independently. 

I was very pleased with the outcome of this activity. The honors students were suggesting ideas as United 
Streaming and the use of flash cards, which they had used in their class. All 19 students eagerly participated and 
wanted to create a final product for one of the SIOP classes to use. There just wasn’t enough class time to do this, 
but I am in the process of working with the Civics teacher and LEP guidance counselor to create a technology club, 
citizenship group, or a combination of the two that will give interested students the opportunity to create a final 
product from one of the ideas. I feel that extensive work on one of these activities will only reinforce what the 
honors students know about the content, which will better prepare them for their EOC. 
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North Carolina Standard Course of Study 
10 th Grade Social Studies Civics and Economics 



Competency Goal 1 


The learner will investigate the foundations of the American political system and 
explore basic values and principles of American democracy. 


1.07 


Evaluate the extent to which the Bill of Rights extended the Constitution. 


Competency Goal 3 


The learner will analyze how state and local government is established by the North 
Carolina Constitution. 


3.06 


Analyze how the Fourteenth Amendment extends the Bill of Rights' protection to 
citizens of a state. 


Competency Goal 10 


The learner will develop, defend, and evaluate positions on issues regarding the 
personal responsibilities of citizens in the American constitutional democracy. 


10.01 


Explain the distinction between personal and civic responsibilities and the tensions 
that may arise between them. 


10.02 


Develop, defend, and evaluate positions on issues regarding diversity in American 
life. 


10.03 


Evaluate the importance of supporting, nurturing, and educating oneself in the United 
States society. 


10.04 


Demonstrate characteristics of effective citizenship. 


10.05 


Describe examples of recurring public problems and issues. 


10.06 


Discuss the consequences and/or benefits of the freedom of economic, legal, and 
political choices. 



Computer Technology Skills 



Competency Goal 3 


The learner will use a variety of technologies to access, analyze, interpret, synthesize, 
apply, and communicate information. 


3.01 


Select and use appropriate technology tools to efficiently collect, analyze, and display 
data. 


3.03 


Use electronic resources for research. 


3.04 


Select and use technological tools for class assignments, projects, and presentations. 
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Abstract 

The primary goal of the study is to analyze learner behaviors during collaborative group work in a digital learning 
community. A qualitative data analysis program, NVivo 2, was used to code the vast amount of online transcripts 
and written interviews based on the grounded theory. Resulting codes were aligned in the four subsystems of 
activity theory and the activation of subsystems by group work phases was investigated. Conflicting factors and 
facilitating factors while college students achieve a common learning goal were identified. At the same time, 
whether the high performing groups show patterns of learner behaviors, conflicting factors, and facilitating factors 
that differ from those of the low performing groups was examined. 

Introduction 

A digital learning community (DLC) is an emerging instructional approach that embraces the characteristics 
of collaborative learning and computer-mediated communication in networked environments. A DLC draws 
attention because it provides students with opportunities to extend their learning experiences by sharing their new 
ideas with, and receiving critical and constructive feedback from, community members (Palloff & Pratt, 2005). 

Also, learning together in a DLC provides chances for students to improve collaboration and communication skills 
that are required on the job (Bennett, 2005). Furthermore, teamwork is another generic skill developed in higher 
education (Candy, Crebert, & O’Leary, 1994). In recognizing the benefits of a DLC, the questions that come to mind 
are “How do learners in a DLC collaborate to achieve a common learning goal?" and “How do we need to design 
and support such nontraditional pedagogies of learning?” 

Despite the promising benefits of collaborative learning, learners experience tensions from mixed feelings of 
wanting to learn independently and a fear of being isolated from the community. Dirkx and Smith (2005) argued that 
these negative experiences are derived from “ambivalence.” A major focus of Computer- Supported Collaborative 
Learning (CSCL) has been providing better ways of understanding learners in communication and collaboration to 
achieve learning goals. Group synergy created by collaboration, however, is not fully explained by CSCL theory and 
still remains as abstraction (Stahl, 2006). 

The direction of research has been geared toward two aspects of CSCL, that is, outcomes and processes. 
Research examined the effectiveness of different tools, techniques, and learner outcomes in collaborative learning. 
On the other hand, the process-oriented research examined socio-cultural factors and learners’ language acts 
(Treleaven, 2004). Activity theory has been used to understand human behaviors in a social context and is one of the 
major theories on which CSCL is based. 

This study attempted to shed light on the process of online collaboration with activity theory. The 
investigation revealed different patterns of learner behaviors during collaborative group work which is aligned with 
the framework of activity theory. In particular, both facilitating and conflicting factors were identified after 
analyzing the data from online transcripts and semi-structured interviews. This study also sought to determine if any 
differences exist between high and low performing groups for both factors in their collaboration activities. 



Theoretical Background 

Activity theory is a philosophical and multi-disciplinary framework to research various forms of human 
behaviors. It has been used as a socio-cultural analytical framework in social contexts with humans and mediators 
(Kuutti, 1996; Jonassen & Rohrer-Murphy, 1999). The applications of activity theory are found in learning (Hung & 
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Wong, 2000; Barab, Barnett, Yamagata-Lynch, Squire, & Keating, 2002; Johri, 2005; Barab, Schatz, & Scheckler, 
2004), human-computer interaction (Kuutti, 1991; Nardi, 1995), instructional design (Jonassen & Rhorer-Murphy, 
1999) and work practices (Engestrom & Middleton, 1996). 

The root of activity theory stems from three historical origins: classical German philosophy from Kant to 
Hegel, the writings of Marx and Engels, and the Soviet Russian cultural-historical psychology of Vygotsky, 
Leont’ev, and Luria (Engestrom, 1987). Activity theory has evolved and reached the third generation. The first 
generation of activity theory stems from the idea of mediation by Vygotsky. The second generation of activity 
theory was derived from Leont’ev’s work. He made distinctions between an automatic operation, an individual 
action, and a collective activity. The third generation of activity theory has expanded to include the activity system 
by Engestrom (1987). 

Activity theory has been further developed as a practical model of human activity, an activity system. An 
activity system contains six interacting components: subjects, objects, tools, rules, division of labor, and community. 
Activity systems are organized to achieve the goals of activities of the activity subsystems (production, exchange, 
distribution, and consumption subsystems) that describe functions, interactions, and relationships between the six 
components. The production subsystem explains how subjects transform the object of the activity system into the 
outcome. The exchange subsystem shows how subjects are constrained by rules and interact with the community in 
accordance with the rules. The distribution subsystem describes how the community defines a division of labor for 
the subject to accomplish the object of the activity system. Lastly, the consumption subsystem shows how the 
subject and the community around the subject collaborate, and also how the community consumes effort from the 
subject (Engestrom, 1987; Jonassen, 2000). 



Research Questions 

The study intended to answer the following questions: 

1 . What are the different patterns of learner behaviors in a digital learning community? 

2. What are the emerging conflicting factors in a digital learning community? 

3. What are the emerging facilitating factors in a digital learning community? 

4. How do the high performing groups differ from the low performing groups in learner behaviors, conflicting 
factors, and facilitating factors? 



Method 



Participants and Setting 

In order to examine collaborators’ behaviors in an online environment, we chose six groups who enrolled in a 
college-level class, titled ‘Information Society and Education,’ in the fall semester of 2006 at a large university in 
Seoul, Korea. Though the class met offline every week, each group of four members also worked independently 
online on the group project. Ranking each group according to performance, we selected three groups to form the 
upper half and the other three to form the lower half. Each group selected an instructional design method, submitted 
a project plan, and implemented an online course. At the end of the semester, each group presented the website they 
implemented to the class. This research used online transcripts of 24 students and semi-structured interviews of 
seven participants. 

Procedures 

The constant comparison method was used to capture real phenomena. As a result, the codes were created from 
the raw data. The qualitative data analysis program, NVivo 2, was used to code consistently the online transcripts 
from six groups for fifteen weeks. The coding scheme was divided into three main categories. The first category was 
learner behaviors that represented the specific behaviors performed by participants. The second category was 
conflicting factors and the third was facilitating factors. The codes in the learner behaviors category was then further 
categorized into the related subsystems (production, distribution, exchange, consumption subsystems) defined in 
activity theory. Upon completion of the coding scheme, the codes and the frequencies of the codes found in both the 
upper and lower halves were compared. 
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Results 



First, the analysis identified 29 different types of learner behaviors. These 29 open codes generated by the 
grounded theory revealed seven themes of learner behaviors in a digital learning community, as shown in Figure 1. 
The seven themes were information seeking (i.e., share material), extraction of relevant information (i.e., summarize 
material), idea generation (i.e., suggest an idea, request an idea, collect ideas, ask questions), co-construction (i.e., 
outline tasks, suggest a meeting, suggest group work, request to do work, etc.), division of tasks (i.e., divide tasks, 
redistribute tasks), making or conforming to rules (i.e., suggest a rule, share template, remind of schedule, raise an 
issue, etc.), and evaluation (i.e., evaluate material, evaluate self or peer work). The two most frequently observed 
categories were information seeking and co-construction. 




Figure 1. Emerging themes of learner behaviors in a DLC 



Those 29 different learner behaviors were aligned with the four subsystems of the activity system based on 
activity theory. Among the four subsystems (production, consumption, distribution, and exchange subsystems) in the 
activity system, the consumption subsystem had eleven different behaviors (i.e., share material, suggest an idea, ask 
questions, etc.) and the highest frequencies of observations. The exchange subsystem had eight different behaviors 
(i.e., suggest a rule, remind of schedule, evaluate peer work, etc.) and the production subsystem showed five 
different behaviors (i.e., modifying material, submitting reports, writing meeting minutes, etc.). The production 
subsystem had more incidents than the exchange subsystem even if it had fewer types of behaviors. Lastly, the 
distribution subsystem showed one type of behavior, dividing tasks, and showed the least number of incidents. 

Figure 2 shows how active each subsystem is according to the project phase. 
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Figure 2. Activated subsystems by group work phases 

Second, six different categories of conflicting factors emerged: inefficiency of work, unfamiliarity, difficulty in 
communication, issues of roles, conflicting schedules, and technical difficulties (Table 2). The factor most 
frequently appeared was inefficiency of work. The reasons of inefficiency included lack of skills, lack of group 
rules, applying inefficient methods, and lack of necessary resources. The next most frequently mentioned factor was 
difficulties of communication in online environment. This factor included uncertainty, nonparticipation, difficulty 
with relationships, and delayed feedback. Other conflicting factors included role-related issues, unfamiliarity, 
schedule conflicts, technical difficulties, etc. When the observed conflicting factors were matched with the 
components in the activity system, the most frequently observed conflicting factors resided between subjects and 
tools components. 

Table 2 

Emerged conflicting factors 



Categories 


Open codes 


CODE 


Definitions 


Inefficiency of 
work 


Lack of skills 


IWLS 


Work is not performed efficiently 
due to a team member's lack of skills 
regarding tools (e.g., Photoshop, 
Flash) required to complete the group 
project 




Applying inefficient 
methods 


IWAI 


Work is not performed efficiently 
due to the fact that a team member 
used inefficient methods to complete 
the group project 




Lack of resources 


IWLR 


Work is not performed efficiently 
due to lack of resources to complete 
the group project 




Lack of group rules 


IWLG 


Work is not performed efficiently 
due to the absence of rules defined 
by team members to complete the 
group project 




Difficulty with finding 
the relevant info 


IWDF 


Work is not performed efficiently 
due to the fact that team members do 
not know how to find relevant 
information 


Unfamiliarity 


Unfamiliarity with 
processes or methods 


UF_UP 


Frustrations due to unfamiliarity with 
how to proceed with the group 
project and with what methods to use 




Unfamiliarity with 
topics or material 


UFTP 


Frustrations due to unfamiliarity 
regarding the project topic or the 
relevant material 
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Difficulty in 
Communi- 
cation 


Uncertainty 


DCUN 


Difficulty in communication caused 
by not understanding what other 
team members meant exactly 




Nonparticipation 


DCNO 


Difficulty in communication due to 
the fact that a team member did not 
participate in a decision making 
process 




Difficulty with 
relationships 


DCDR 


Difficulty in communication due to 
discomfort among other team 
members especially when there are 
age differences 




Delayed feedback 


DCDF 


Difficulty in communication caused 
by the nature of asynchronicity of 
online communication 


Issues of 
roles 


Work delays 


IRWD 


Issues of roles due to the fact that a 
team member does not complete 
one's assigned work on time 




Varying levels of 
contribution 


IRVC 


Issues of roles due to the fact that a 
team member recognizes inequality 
of efforts made by each team 
member 




Issues of role 


IRRA 


Issues of roles due to the fact that 




assignment 




roles were not assigned equally. 


Conflicting 

schedules 


Conflicts with other 
personal commitments 


CS_PC 


Conflicting schedules among team 
members due to jobs, part-time work, 
or other personal commitments. 




Conflicts with other 
subjects/exams 


CSEX 


Conflicting schedules among team 
members due to other subjects or 








exams 


Technical 

difficulties 


System issues 


TDSY 


Issues with sharing files due to the 
learning management system 




Corrupted or 
incompatible files 


TDFL 


Issues with sharing files due to 
corrupted files, or incompatibilities 
between different versions of 
software, etc. 



Third, the analysis revealed five facilitating factors: efficiency of work, effective communication, the 
competence of team members, group cohesiveness, and goal orientation (Table 3). The most frequently observed 
factor was group cohesiveness. This category was composed of intimacy, a sense of community, and encouraging 
others. When the observed facilitating factors were placed in relevant components in the activity system, the most 
frequently observed facilitating factors were located between subjects and community components. 

Lastly, comparing the high performing groups with the low performing groups, no difference was found in 
terms of types of learner behaviors. One evident difference was that the high performing groups revealed about 40% 
more of such incidents. When the learner behavior codes were aligned in the activity system, the consumption 
subsystem was the most highly activated subsystem, followed by the production, exchange, and distribution 
subsystems, in that order. A salient difference between the high and low performing groups was that the frequency 
of the behaviors in the consumption subsystem was in decline at the project completion phase for the high 
performing groups, whereas that of the low performing groups was increasing. As for facilitating and conflicting 
factors, the high performing groups revealed more incidents of conflicting factors and fewer incidents of facilitating 
factors. 
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Table 3 

Em erged facilitating factors 



Categories 


Open codes 


CODE 


Definitions 


Efficiency of 
work 


Applying 
efficient methods 


EW_AE 


Work is performed efficiently because 
team members use efficient methods to 
do the group project 




Conforming to 
rules 


EWCR 


Work is performed efficiently because 
team members follow the rules defined 
by the team to do the group project 




Efficient role 
assignment 


EWFR 


Work is performed efficiently due to 
efficient role assignment based on team 
members' strengths 


Effective 

communication 


Timely decision 
making 


ECDM 


Team members make a decision 
through responsive communication 




Honesty 


ECHO 


Team members talk straight regarding 
the way or the quality of peer work 




Proactiveness 


ECPR 


Team members show eagerness or take 
an initiative in communication. 


Competency of 
team members 


Responsibility 


CTRE 


Team members are responsible for the 
assigned task or the project overall 




Competency of 
tools 


CTCT 


Team members are competent in using 
tools required to complete the group 
project 




Previous 

experience 


CTPE 


Team members have previous 
experience or prior knowledge to do the 
group project 


Group 


Intimacy 


GC IN 


Team members feel close to each other 


cohesiveness 


Sense of 
community 


GCSC 


Team members have a sense of 
community, referring to the group as 
'we', 'us', or 'our' 




Encouraging 

others 


GCEO 


Team members encourage each other to 
keep up the good work 


Goal oriented 


Sense of 
competition 


GOSC 


Team members feel a sense of 
competition with other teams 
(performance goal) 




Excellent 

outcomes 


GOEX 


Team members strive to create 
excellent outcomes (mastery goal) 



Discussion and Implications 

This study produced a coding scheme that can be used to analyze learners’ collaborative behaviors. To generate 
a coding scheme we have used a mixed approach that is grounded in empirical data and theory based (activity 
theory). The results of this study showed how production, exchange, distribution, and consumption subsystems were 
activated during collaborative work. Each subsystem can be considered as a learning space in a digital learning 
community. This study revealed that the consumption subsystem was the most highly activated subsystem 
throughout the project phases. DLC design should be able to support learner behaviors identified in the consumption 
subsystem. Likewise, when learners engage in the production subsystem, necessary tools or artifacts should be 
available in the DLC environment. In addition, rules govern the learner behaviors during group work. DLC 
designers should take into account rules that facilitate effective and productive learning. Lastly, division of labor 
should be designed to ensure both individual accountability and optimal interdependency among team members. 

Based on the conflicting factors found in this study, some implications can be made. First, there is a need for 
different implementation strategies depending on the nature of conflicting factors. It was evident that some of the 
conflicting factors are not as harmful but can serve as an alert for interventions. The fact that the high performing 
groups revealed higher incidents of conflicting factors supports Engestrom’s point (2002) that development occurs 
when contradictions are overcome. Apparently, other types of conflicting factors are harmful and can lead to 
negative learning experiences or deficient learning outcomes. These include work delays, varying levels of 
contribution, issues of role assignments, and nonparticipation. An instructor or a facilitator should proactively 
monitor and intervene by using them as indicators. 
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Recommendations for Future Research 



Based on the results of this study, the following future research is suggested. First, the findings of the study 
have drawn a few implications in providing an effective DLC learning environment. Those DLC design implications 
are required to be validated and refined through design-based research. Second, the study implied two different 
kinds of conflicting factors. This interpretation requires in-depth research regarding specific conflicting factors and 
how these factors can affect student achievement or learning processes in a DLC. Third, group cohesiveness was the 
most frequently observed category of facilitating factors. In this study, group cohesiveness is not directly related to 
group performance. Further research on group cohesiveness and performance would be beneficial. Lastly, the digital 
learning community in this study was formed rather involuntarily due to the nature of higher education. Future 
research could replicate this study in an autonomously formed learning community to determine if it reveals 
different patterns of learner behaviors, conflicting factors, and facilitating factors during group work. 
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Introduction 

Second Life is a free, online virtual reality world that has averaged over 1 million new users worldwide in each of its 
first three years. What is all the fuss about and what are the implications for use in education? Our presentation will 
involve a live demonstration of this virtual reality environment and present research whose initial findings suggest 
that several positive benefits are associated with its use in our graduate level distance learning program. 

The immediate thought that comes to mind when one hears the words “Second Life” is a part sarcastic, part tired 
thought of, “I already have too much to deal with in my first life!” This is what many of us were thinking as we 
trooped into a presentation and demonstration of Second Life. Expecting another contribution to the social 
collaboration tools that are currently redefining the Web, we were greeted instead by a game-like 3D virtual reality 
world where users or “residents” move around and experience a digital environment with a digital-self or avatar. 
Several members of our faculty became intrigued with the possibilities such a free, easily accessible environment 
presented. We thought: could this be more than just another trivial addition to the Web? 

Given the implications for increasing our online presence and providing innovative interactions and support to our 
sizeable distance learning student population, many of us decided to take a moment out of our busy “first lives” to 
take a closer look. Should my digital self be tall or short? Fat or skinny? Muscular or slim? Blond or lime green 
hair? Five feet or seven feet tall? Faced with such engaging decisions about usually immutable physical 
characteristics in a game-like environment, where decisions are immediately rendered in 3D digital images before 
your eyes, it did not take long for many of us to become absorbed in this new, expansive, beautifully rendered 
virtual world. 

Second Life has garnered quite a bit of notoriety since its launch in 2003. As of February 2007 there were over 3 
million residents worldwide (Retrieved February 13, 2007 from http://secondlife.com/whatis/). IBM is one of many 
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Fortune 500 companies investing large amounts of resources in establishing a commercial presence (Kirpatrick, 
2007) and, in the academic world, seemingly every week there is a new article on the impact this virtual world is 
having on the real world around it. Why all of the excitement? 

Present day society with its affordable computing and high bandwidth infrastructure has made great strides in 
catching up with providing the necessary memory, processing speed, and connectivity to experience the full effect of 
virtual reality technology. Second Life defines itself as, “An online society within a 3D world, where users can 
explore, build, socialize, and participate in their own economy” (Retrieved February 13, 2007 from 
www.secondlife.com ). Some say that, similar to real life (referred to simply as RL in Second Life) there does not 
appear to be a real goal or end point and although it is initially free, to actually participate or own any land you have 
to pay for it (Diski, 2007). But proponents say that this environment adheres to one of the more desirable and 
addicting aspects of the computer gaming industry - having to figure out what to do in the first place (Wagner, 

2007). In addition, virtual worlds today usually offer the opportunity to explore in an online, socially connected 
exchange with thousands of other simultaneous users in an “immersive environment” that is similar, yet different in 
many significant ways than the real world (Bixler, 2007; Dede, Clark, Ketelhut, Nelson, and Bowman, 2005). 

The literature has found that the use of avatars are particularly effective when addressing motivation-related 
outcomes, especially as it is associated with a learner’s self-efficacy or confidence that one can accomplish or learn 
some identified task (Park, 2007; Baylor & Kim, 2004); particularly engaging to users is the ability to manipulate an 
avatar’s ethnicity, gender, and other physical attributes (Park, 2007), which is a major component of the Second Life 
experience. Educational purposes, however, focused primarily on two primary aspects of virtual reality: an 
immersive environment and social interactions (Bixler, 2007; Dede, Clark, Ketelhut, Nelson, and Bowman, 2005). 

What was it that a graduate faculty at a mid-sized southeastern university saw in Second Life that so engaged us? 

One of the immediate ways of using this technology was to ask students to participate in this immersive, socially 
interactive environment for our distance courses, which typically are a blend of face-to-face, television, and course 
Web site. One course has used this environment to provide its “face-to-face” lecture in a virtual, private “sky box” 
that brings students from three campuses divided by real constraints of distance together into one “class room.” We 
also began holding “virtual” office hours where students at a distance in particular expressed the desire and ability to 
just “drop” in to talk to a faculty member just like local students can. Second Life has also served as an ever present, 
free resource for students to interact with and evaluate a living example of an impressive use of multimedia 
technology. 

A review of the relatively young body of literature on the educational impact of Second Life suggests that the 
primary focus has been on the multifaceted ways people are currently using it, especially in the business sector, but 
with no focus on learning outcomes or student attitudes, especially for non-traditional school aged students 
represented by graduate students. Although our research is in its preliminary stages, we have used a mixed-method 
approach to address two primary questions that we feel will make a significant contribution to the literature: 

1 . In what ways has the use of Second Life impacted learning and instruction in distance learning programs? 

2. Flow usable is Second Life in terms of utility and general ease-of-use for students and faculty? 

Data collection involved natural observation, qualitative interviews, user surveys, and content analysis of discussion 
board transcripts. 



Method 



Participants and Instrumentation 

All 18 participants in the study are samples of convenience. The study’s sample included three faculty members 
from a mid-size southeastern university graduate program that used Second Life as part of their blended (n=3) and 
online courses (n=l), a Second Life instructor who teaches virtual seminars, nine graduate students who participated 
in a course featuring Second Life as a primary mode of course interaction, and four students who attended a series of 
virtual seminars in Second Life (n=4). The data collection period was from January 2007 to October 2007. 
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Faculty (n=2). Two of the authors of this study are tenure track faculty members who teach traditional and distance 
graduate courses. Each documented how Second Life was used in their courses, how students reacted, and general 
thoughts about overall utility and potential use in distance learning. 

Second Life Instructor (n=l). The Second Life Instructor teaches several virtual seminars in Second Life and asked 
eight of her students a set of six questions including, “How quickly did you pick up the skills necessary to be 
successful in the class?”, “What are the positive and negative aspects of learning in a virtual environment?”, and 
“What real life lessons and/or skills were learned from your experience in a virtual environment?” A content 
analysis was conducted of the four student responses that were received. 

Graduate Students (n=9). Students in a face-to-face course studying online collaboration tools used, examined, and 
discussed Second Life and its potential use for educational purposes. A content analysis was conducted of student 
discussion board transcripts which took place over a one week period (January 28-February 4 th , 2007). 

Results 



The Wow Factor 

Second Life is a free to use three dimensional world that adheres to one of the paramount standards of usability 
which is user control or “designing a product so that the extent to which the user has control over the actions taken 
by the product and the state that the product is in is maximized” (Jordan, 1998). Becoming a member of Second Life 
is relatively easy and painless and the process and engagement involved in creating a virtual self (referred to as an 
avatar in Second Life) is addicting and relatively limitless. 

Our program purchased virtual property within Cybrary City, an island where libraries and information resources are 
made available in Second Life and each of our t'ac Lilly created their own avatar. A photo of the faculty virtual avatars 
in front of our virtual building is shown in Figure 1. 




Figure 1 - Faculty Avatars and Department's Virtual Building 
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Presented with the ability to freely change an avatar’s features, clothing, and virtual possessions, our faculty quickly 
embraced Second Life as a vibrant place for exploration and social collaboration with tremendous possibilities for 
our graduate program: Social collaboration amongst faculty and students, three dimensional information seeking 
both for reference and program marketing purposes, and classroom presentations and work groups in small group 
settings. 

One of the biggest problems with attempting to conduct a class within a public building is the potential for anyone to 
interrupt class, which in the real world is clearly inappropriate but in the virtual world is a facet of understanding the 
dynamics of the SL social situation. In order to protect against this, one of our faculty members created a private 
virtual skybox that can only be accessed through private invitation. Figure 2 shows an avatar seated in our virtual 
classroom in the sky. 




Figure 2 - Private virtual classroom in the sky 



Faced with a blended course involving over 80 students across three campuses statewide, one of the authors of the 
study felt Second Life would be an ideal opportunity to build stronger collaboration between students from different 
campuses. The use of Second Life, however, turned out not to be that simple. 

Second Life Meets Academic Reality 

To enter the world of Second Life the computer you are using must have the Second Life software installed and 
have the computing hardware in terms of RAM, processing speed, and graphics card necessary to run this robust 
software. Prepared to present to the entire class from the classroom teaching station, the author realized that the 
software had not been preinstalled by technical support and impromptu installations were not allowed without an 
administrator password. The long awaited preview of Second Life had to wait a little longer. 

The following week Second Life was successfully previewed in class and all students were informed that in 
conjunction to the author’s on campus office hours, virtual office hours would also be available in Second Life. In 
addition, the author brought up the possibility of conducting a full class session completely in Second Life. While 
the feedback from students were predominately positive regarding the “wow factors” of being able to create a virtual 
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self, fly and move freely around a three dimensional virtual world, and meet new people in a uncontrolled 
environment, the overall course outcomes were less favorable. No class meeting was ever held in Second Life 
because a small, yet significant minority of students could not access it through their older computers and when 
these students approached their campus academic libraries all were told that Second Life was not installed on the 
library computers and could not be without a strong rationale from the instructor. Due to time constraints, the author 
did not attempt to articulate this rationale as it would have involved engaging with three university libraries 
simultaneously. The author’s virtual office hours also mirrored the overall usage of physical office hours, which 
was not utilized at all and a surprise since the availability for individual student contact had been increased by over 
50% by making synchronous office hours available to the two distance sections 1 . The following semester, one of 
the authors decided to try again and offer virtual office hours via Second Life to an entirely online course. This 
attempt met a similar fate where no student took advantage of this opportunity to “drop in” to meet with the 
instructor during posted virtual office hours. 

A second author of this study had students engage with Second Life as a focus of a class as it provided an 
environment rich in learning resources and an opportunity to conduct class and discussion in an online environment. 
Students were introduced to Second Life in a face-to-face class where they were given the opportunity to interact 
with the instructors and other students in making their initial foray into the alternative world of Second Life. Many 
of the students were apprehensive in developing their avatars and making initial contact with other participants in 
this environment. They came together as a class in the online environment to receive instruction and explore 
learning opportunities. Instructors planned and conducted a journey with the students bringing the class to specific 
locations and then discussing the uses of the particular “landmarks” or locations within Second Life. Students were 
able to bookmark landmarks for further exploration and were able to ask questions of other students and instructors 
as the tour progressed. 

Class was also held in Second Life with mixed results. The idea of having a real time chat in the online environment 
was beneficial but the use of a virtual physical space may have been unnecessary for the content of the particular 
class. The use of PowerPoint and notes within the virtual classroom, however, added a more dynamic element to the 
class and made better use of the tools that Second Life offers to an instructor. The class offered an opportunity for 
both instructors and students to operate in a virtual environment both synchronously and asynchronously; exploring 
the resources and meeting with each other. Problems that arose centered on the fear of technology and the learning 
curve of some students but everyone in the class was able to overcome their hesitations and participate in the virtual 
classes with a minimum of difficulty. 

Student comments varied widely in their perceptions of this first experience. All nine students found Second Life to 
have uniquely positive aspects for educational purposes: it connects users to a wide array of information resources, it 
provides “residents” with the ability to fly and transport one’s avatar from one place to another, it provides a 
generally “stimulating,” immersive, and interactive learning environment, it allows users to virtually “experience” 
information as opposed to just reading text, it socially connects people and organizations from different parts of the 
country and world “who you would probably never meet in real life,” it is a cost effective way to meeting virtually 
as opposed to actually “face-to-face” with other people, it provides opportunities for virtual field trips and simulated 
experiences, and, in general, the 3D world is much more conducive to an online educational learning environment 
then traditional flat, one dimensional text-driven digital displays. 

There were also a number of negative aspects. First and foremost, were the technical issues involving the need for 
robust computing and connectivity - one student reported having his computer’s graphics card malfunction. In 
addition, students ran into the general prohibition of the use of such software 2 in public computing environments 
where students tried to take advantage of more robust hardware to use Second Life. Another primary point brought 
up was the potential of “wasting time” and challenge of “time-on-task” especially when attempting to teach a large 
number of students. An additional issue was that while the “uncontrolled” environment of Second Life can prove 
educationally stimulating at the same time it does make it a challenge to manage the learning experience and ensure 



1 The course included three sections, one face-to-face local section, and two distance sections. 

2 Second Life is not a Web site that you connect to but instead requires you download its client software directly 
onto the computer you are trying to connect with. This causes major issues in attempting to access Second Life in 
public computing environments such as public or academic libraries because most prohibit downloading of 
unauthorized software. 
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consistency in student experiences. Finally, while online collaboration can be quite enjoyable and successful at 
times, students brought up the concern of the coldness and artificial reality associated with technology as an explicit 
danger, especially when faced with the thought of entirely replacing real human interaction with a digital, virtual 
world. 

Providing Instruction in Second Life 

Two applied examples of how Second Life is currently being used for instruction are continuing education Virtual 
Librarianship courses offered by the University of Illinois at Urbana-Champaign and a technology course taught at 
San Jose State University. 

The University of Illinois at Urbana-Champaign and the Illinois Alliance Library System has made use of virtual 
adjuncts to teach continuing education, no-credit, six week courses in Second Life. Five courses have been offered, 
two in the summer 2007 and three in fall 2007, with an average class size of 20-30 students per course. These 
classes are taught by librarians in Second Life and have covered such topics as Intermediate Virtual World 
Librarianship, Virtual World Librarianship in Second Life, and Libraries and Immersive Learning in 3D Virtual 
Environments. Table 1 below shows the general breakdown of one of these courses: 

Table 1 - Virtual Course Overview 

Virtual World Librarianship in Second Life Course Overview 



Week One: Introduction to Libraries in Virtual Worlds 
Week Two: Second Life 101 

Week Three: Collections, Resources, and Exhibits in Virtual Environment 
Week Four: Reference and Information Services in Virtual Worlds 
Week Five: Managing and Working in a Virtual Library or Department 
Week Six: Skills Needed by 21st Century Librarians in Virtual Worlds 

Jeremy Kemp at San Jose State University taught a section of Information Technology Tools and Applications in 
summer 2007 that included bringing students into Second Life and having them work on projects with Second Life 
librarians to illustrate examples of how the software they were studying was being applied as well as to locate virtual 
resources around designing advanced Web applications using technology such as dynamic HTML and XML and 
Web programming languages such as XSLT, JavaScript, JSP, Perl, PHP, MYSQL, and ASP. 

One of authors of this study works for Second Life as an Assistant Director/Operations Manager for the Alliance 
Library System's Info Island Archipeligio. As part of the study, she asked her students about Second Life. Two 
students worked with her on a business/tourism project called Virtual Morocco and answered her questions in 
Second Life. A second pair of students she worked with in a class being taught at a local university answered 
questions via email. 

In response to the question, “How quickly did you pick up the skills necessary to be successful in the class?” all four 
students responded that they had easily picked up the needed skills quickly and felt confident about using Second 
Life within two to three weeks. The more experienced students initially were called upon to do more advanced tasks 
in Second life and were taught building and scripting skills, which they learned with relative ease. Students felt that 
the most positive aspects of learning in a virtual environment were that it was: Collaborative, asynchronous, 
involved visual and kinetic learning, and provided opportunities for interaction with others inside and outside of the 
class. The major negative aspects of learning in a virtual environment involved: Problems with technology, high end 
requirements, Second Life downtime for maintenance, use of traditional classroom teaching methods while in the 
virtual world, problems encountered from disputative Second Life residents, and learning to communicate textually 
requiring instantaneously formulating answers and responses and fast typing skills 3 . In terms of the most valuable 
lessons learned from working with Second Life the students listed the following: Time management, self reliance, 
leadership, teaching, social networking, graphic arts and design, experience working with virtual groups, and 
experience with virtual reference. 



3 Second Life has added a voice component that should help alleviate this requirement. 
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Discussion and Conclusion 



Teaching and Learning Opportunities in Second Life 

The use of Second Life in graduate education and in teaching and learning environments in general are not limited to 
distance learning. As a free resource that in the past would have been cost prohibitive, Second Life represents a 
relatively open access, immersive virtual 3D learning environment that has many potential uses in teaching and 
learning environments. Based on the results of our study and review of the literature, here are some of the major 
teaching and learning opportunities Second Life represents: 

Placing an Avatar with a Name - In the past, and for the most part even today, a typical synchronous online 
interaction in distance education involves text-based chat. Users are defined by their user name and communicate by 
typing text based messages. Second Life, ironically also uses text-based chat as the primary method for 
communication. Changing the communication environment from a flat, text based one dimensional interface to a 
rich, multi-color, 3D environment that allows users to represent themselves with virtual avatars or representations of 
themselves. As student isolation is one of the major problems associated with distance education, the ability to 
“place an avatar with a name” and add the aspects of non-verbal representations of personality and feeling, has 
significant implications for social collaboration and communication in online environments. While sharing the major 
benefit of online, synchronous communication, which is to not have to meet face-to-face at a specific physical 
location. Second Life adds many additional layers through virtual representations of physical characteristics and 
communication to this online interaction. 

Synchronous Virtual Information Seeking - Unlike navigating a Web site, in Second Life you have the 
opportunity to interact with both static information and synchronously with other users. The concept of adding a 
synchronous, three-dimensional aspect to online information seeking is one of the aspects of Second Life that has 
captured the attention of the corporate world. The same benefits apply to teaching and learning environments where 
students can both access digital content in Second Life while at the same time engaging in a discussion with a 
student or instructor. In addition, other users through their virtual avatars serve as additional resources that otherwise 
would not available. For example, at Cybrary Island, information science buildings from across the world are housed 
together ranging from the US to Europe to Asia. Interaction amongst instructors and students are frequent and, 
engaging in discussions around mutual topics of interest that is truly global in nature, has many possibilities. 

A Virtual World Allows for Virtual Experiences and Resources - In Second Life you have the ability to 
perform tasks, create objects and environments, and interact with people and objects that you cannot do in real life. 
For example, you can fly, teleport, and generally control and customize your avatar in unlimited ways. Outside of 
the social consequences of being able to create a digital self, creating virtual learning environments or interacting 
with existing content rich ones is where Second Life excels. Although like most online instruction, planning and 
preparation is necessary to find relevant resources and to ensure the integrity and effectiveness of instruction. 

Second Life can be used as virtual field trips so that students can not only find relevant resources but also potentially 
have the opportunity to see and interact with 3D digital information as well as other users as information resources. 

Technology as a Nexus Point - The Internet and the Web connect people and information together. 
Traditionally this is done through digital information usually in the form of text, graphics, video, and document/file 
exchanges. In distance education, television/video conferencing, Web cams and Web casts, threaded discussion 
boards, and synchronous chat are the predominate mediums in which collaboration is achieved in one-to-many 
instmctor-to-student educational environments. Web conferencing software, which integrates Web video, file 
sharing, and synchronous chat is highly desirable in the delivery of distance education but is extremely costly and 
usually involves dedicated internal resources and support by the distance program. Second Life offers a potential 
nexus point in which distance programs can utilize this virtual world at little or no cost for its instructors and 
students. Social connection and collaboration in many ways is the easiest to attain in Second Life and at the same 
time helps meet an essential need for distance students. 
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Challenges of Using Second Life 

Technology - Although broadband connectivity and computing power continues to become more robust 
and inexpensive, to operate in the vibrant digital world of Second Life takes high end computing power. This poses 
a significant barrier equally for instructors and students as computers that are only a couple of years old may not 
have a high end enough graphic card able to handle Second Life’s robust graphics. With the increasing popularity of 
Second Life this poses a unique digital divide that previously did not exist. In addition, as Second Life requires a 
software client download, it is difficult to use public computers thereby severely limiting its overall portability. 

Allocation of Time and Resources - Second Life is time intensive for many reasons. First, it is highly 
addicting because of the large number of options and overall control it provides users, especially as a user’s avatar is 
the virtual representation of that user and therefore usually garners careful attention to such details as height, weight, 
hair color, skin color, general physical features, and certainly clothes. Second, learning how to operate in this virtual 
world, although relatively intuitive, takes some acclimation and experimentation. Third, is the time and resources it 
takes to create a digital instructional environment (to have personal space you must purchase virtual property), 
which can be substantial. Training for students must also be factored in. Fourth, is the amount of set up time and 
technical support required to ensure all students are successfully able to access the high end computing resources 
necessary to interact with Second Life. Lastly, in terms of instructional design, attempting to use standard classroom 
teaching methods in a digital world usually does not translate very well, especially if it is almost completely chat 
based. The addition of voice communication will help mitigate this problem. 

Uncontrolled Learning Environments - A user’s avatar is called a resident in Second Life. Unfortunately, a 
majority of residents are not in Second Life for educational purposes and therefore, when trying to provide 
instruction there is always a chance that an instructor or students will be interrupted. In addition, similar to the 
impact laptop driven wireless computing has in classroom settings, the opportunity for students to lose interest or 
become easily distracted is considerable. 



Conclusion 

Distance learning programs use course management systems to organize and deliver content as well as manage 
course functions. Second Life represents another medium in which to engage and collaborate with students or have 
students engage with peers. At a cursory level, this unique digital virtual world allows for more complex social 
interaction as it introduces personal expression through virtual avatars that bring virtual non-verbal communication 
and interaction. At more complex, well planned levels. Second Life becomes a vibrant learning environment where 
self-exploration, virtual tours, and access to a diverse set of users are easily accessible and available. 

Distance learning is a great way to allow students who are scattered geographically to take advantage of a class that 
does not require face-to-face classroom instruction. While the technology is in place to facilitate distance learning, 
the actual class experiences of traditional distance learners can be less than satisfying for the student and the 
instructor. From the student's perspective it is more difficult to form relationships with others in the class and the 
instructor. While the materials may be fully covered and the assignments completed, graded and returned 
electronically the overall experience lacks the kind of connection students feel in a standard class. Second Life, and 
other virtual environments help bridge this gap by giving students the feeling of being present with other classmates 
and with the instructor. They have the ability to talk to each other during class in private instant messaging and to 
ask questions of the instructor as the questions arise naturally out of the lesson plan. Although this is still a form of 
electronic communication, it creates a shared experience for the students and instructor. It is difficult to explain the 
connection, but one of the Linden Labs employees, Pathfinder Linden, calls this connection "emotional bandwidth." 
It adds an element of depth that cannot be achieved in a standard distance learning situation. 

Another way that Second Life and other virtual environments will impact distance learning is to enable a greater 
variety of classes to be taught. In a 3D environment, instructors may now demonstrate instruction as well as lecture 
and students may work together to create projects within the virtual world. In addition, with access to a greater 
number of global residents the opportunity for synergy and collaboration expands. The technology of 3D virtual 
worlds is being expanded and improved and many private and public organizations are creating virtual spaces that 
someday will be tied together much like web pages of today. These early explorations of education in Second Life 
will help to build a foundation for virtual education of the future. 



82 




References 



Baylor, A., Kim, Y. (2004). Pedagogical Agent Design: The Impact of Agent Realism, 

Gender, Ethnicity’, and Instructional Role as cited in Intelligent Tutoring Systems, 

Heidelberg: Springer Berlin. 

Bixler, B. (2007). Second Life educational possibilities and considerations. Retrieved on 
February 14, 2007 from Penn State Virtual Worlds at 
http://ets.tlt.psu.edu/virtualworlds/7page id=67 . 

Dede, C., Clarke, J., Ketelhut, D., et al. (2005). Fostering Motivation, Learning, 
and Transfer in Multi-User Virtual Environments. American Educational 
Research Association 2005 annual conference proceedings. 

Diski, J. (2007). Jowls are available. London Review of Books, Vol. 29, No. 3, dated 8 February 2007. 

Kirkpatrick, D. (2007). It’s not a game. The 3-D online experience Second Life is a hit 

with users. IBM’s Sam Palmisano and other tech leaders think it could be a gold 
mine. Fortune Magazine, February 5, 2007 edition. 

LaPlante, A. (2007). ‘Second Life’ lessons: Cisco, IBM pace corporate push into virtual 
worlds. Information Week, February 3, 2007. 

Park, S. (2007). Techonline: Design and development of conversational pedagogical 

agents embedded learning environment to promote students ’ learning and interest 
in Information Technology fields. Unpublished grant proposal to NEF grant 
program. 

Second Life Web site. What is Second Life? Retrieved February 13, 2007 from 
www.secondlife.com . 

Wagner, M. (2007). Second Life is hard to use - is that a bug or a feature? Information 
Week, January 31, 2007 edition. 



83 



The Design of Collaboration in the Virtual Classroom 



Ana-Paula Correia 
N03 1 Lagomarcino Hall 

Center for Technology in Learning and Teaching 
Iowa State University 
Ames, IA 50011, USA 
acorreia@iastate.edu 

Niki Davis 

N108 Lagomarcino Hall 

Center for Technology in Learning and Teaching 
Iowa State University 
Ames, IA 50011, USA 
nedavis@iastate.edu 

Issues with Online Collaboration 

Even though college-level online education has steadily been growing with a potential to promote 
democratization and the advancement of scholarship of teaching (Larramendy-Joerns and Leinhardt, 2006), distance 
education is often the weakest link in supporting problem-solving and metacognition (McLoughlin and 
Hollingworth, 2001). Having technology and a communication infrastructure seems not to be enough to facilitate 
meaningful online dialogue conducive to learning. Improving online students’ engagement in course content and 
class community through effective design of collaboration is the focus of this paper. 

Learning in online communities evolves as a succession of changing actions; it requires: (a) constant 
change of position in an endless variety of patterns (like in a kaleidoscope) and (b) exchange of roles between 
instructor and students. Online discussions are essential as a way to facilitate learning. However, research shows that 
students may not truly participate in online discussions. Their participation may become a superficial and a 
mandatory exchange of information, which betrays the learning goals behind the use of discussion forums as part of 
online courses. 



The Design of Online Collaboration: An Exploratory Study 

This study investigates the design of collaboration among students in two online courses. These were 
supported by the learning management and delivery system, WebCT. The goal was to analyze the design of three 
types of discussion formats and assess their impact on students’ participation in online discussions. 

Context and Participants 

This study took place in a large research university in the Midwestern United States with a recognized 
leading program of instructional technology in teacher education (Davis, 2003). This same university offers a 
Masters of Education at-a-distance in Curriculum and Instructional Technology (http://ctlt.iastate.edu/~citmed/). 

This graduate degree program was designed to meet the needs of teachers and other educational practitioners 
seeking leadership positions for in fusing technology into teaching and learning, who were widely spread across rural 
Iowa. 

A total of 3 1 students participated in this study. Eighty percent of the students were in-service teachers and 
members of the first two cohorts of the Masters of Education at-a-distance. They were elementary and secondary 
level teachers working in several schools across the state of Iowa, and teaching a variety of topics (e.g., History, 
Math, Science, English, Art and Technology). The remaining students were traditional non-cohort students in the 
Instructional Technology graduate program. Since some of these students had full-time jobs that required a 
significant time commitment, these online courses were especially attractive to them. 
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WebCT was organized with a focus on resource sharing and discussion. Different discussion forums were 
created to facilitate the sharing of experiences and knowledge. In each of the online courses, students experienced 
three different collaboration designs. The formats were: (a) large group discussion facilitated by the instructor, (b) 
small group discussion facilitated by the instructor, and (c) small group discussion facilitated by peers. An 
anonymous course feedback discussion area was created as a way for students to offer continuous feedback on the 
course. 



Data Collection Methods 

Data were collected through analysis of discussion threads posted in the WebCT Discussion area, course 
materials, and responses to a 13-item questionnaire. Ten items out of the 13 were Likert-scale items based on a five- 
point scale from “Strongly Disagree” to “Strongly Agree.” These questions addressed level of participation in the 
discussions, quality of the feedback from classmates, use of critical analysis skills, team members’ engagement, and 
learning by sharing reflections. Three open-ended questions were part of the questionnaire as well. These dealt with 
preferred discussion formats and reasons for such preferences. 

Major Findings 

The analysis of data identified a series of factors related to whether the instructor and/or a peer facilitated 
the group and the content of the discussion forum, as well as its role in students’ engagement. The following 
paragraphs describe the major findings of this study. 

Most respondents agreed that their level of participation in the discussions were high, but they showed 
some indecision regarding the quality of the discussion. Most of them reported not having learned much from the 
large group discussion, reason why they were not involved. There were several reasons related to this situation. 
Students felt lost and overwhelmed in the large group discussion and they also described a low involvement because 
of the other numerous assignments required by the courses. Overall, students were undecided regarding advancing 
their critical analysis skills by participating in the discussions. Divergent responses on team members’ engagement 
in small group discussion were found. 

Most students agreed that their participation efforts in the weekly discussions (large or small group) were 
not extraordinary and that the level of interaction in the large group discussions was also low. Sharing reflective 
summaries in the group discussions was not perceived as a learning opportunity. 

When asked on which discussion formats worked better for them, 62% of the students answered “small 
group discussion facilitated by peers.” This format offered concrete questions and gave the students and option to 
choose the ones to answer. Interaction was higher when discussions were introduced as a topic either open-ended or 
related to practice. One of the respondents explained why this was his favorite discussion format: “Because I can 
bring my own background knowledge & make it relevant to me.” 

All participants stated that “large group discussion facilitated by the instructor” was the discussion 
formats they found less useful, meaning less conducive to class participation. Following are some of the students’ 
comments: 

• “It was not compelling & everyone answers the same thing.” 

• “Questions ‘straight from the book’ usually try to elicit a specific answer - the first couple of 
people who answered said what there was to say about question and the rest of the responses were 
tough to create...” 

• “Didn’t allow for real life application & discussion.” 

• “Not the way discussions work & my brain works.” 

• “Not useful. Too abstract.” 

• “The basic questions get answered in the 1 st few postings. Then no one has much more to say 
other than a paraphrase of previous postings.” 

• “Not applicable to me - motivated only by it being a class requirement.” 

• “While some of these topics were of importance, I found myself just commenting in the discussion 
because it was a requirement for participation. The large group discussions were harder to follow 
with so many class members and I felt that at times people were ignored in them. Part of this was 
because we did have such as large class and often someone would state what you wanted to say 
before you got to it, then you had to stretch to figure out something else to say so you were an 
active participant.” 
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Even so, the anonymous course feedback in WebCT became one of the most popular forums for the 
courses large group discussion. In one of the courses, a vivid discussion in this forum voiced the students’ opinions 
on how to design the collaborative forums. The first posting in this discussion read as follows: 



Message no. 1279[Branch from no. 239] 

Posted by Anonymous on Monday, September 19, 2005 7:52am 
Subject: Re: A few comments 

I am mystified — and disappointed — that many "participants" haven't treated the on-line discussions as a 
chance to interact with their classmates and actually learn by DISCUSSING! It is a DISCUSSION forum. It 
is not supposed to be a short answer essay where you simply turn in your answers by the deadline. There is 
not much participation in the chat sessions either, despite the instructor’s efforts to accommodate those who 
complained about the scheduling. 



This posting initiated an intense conversation among the students on the format of the course weekly 
discussions. It prompted a sequence of replies. The following excerpts illustrate the content of such debate. 



Message no. 362 

Posted by Anonymous on Monday, September 19, 2005 10:1 1pm 
Subject: Discussion Questions 

I would appreciate our discussion questions being on more of an analytical or reflective level so that it is 
easier to think of the topics from various points of view and situations. 

So far, they have seemed so straightforward that within the first few postings, we basically have the 
questions answered and those of us that don't post within the first few days are left with nothing to add. 



Message no. 419[Branch from no. 412] 

Posted by Anonymous on Wednesday, September 21, 2005 4:33pm 
Subject: Re: Discussion Questions 

Perhaps we should be challenged as students to make that leap ourselves after we have answered the factual 
elements of the discussions. 



Message no. 436[Branch from no. 419] 

Posted by Anonymous on Wednesday, September 21, 2005 9:15pm 
Subject: Re: Discussion Questions 

I believe some already have, but might be worried because they haven't specifically answered the question. 
But, to do otherwise might make their comments seem redundant with all the others. 



Summary of the Findings 

Small group as opposed to large group discussion was the preferred format. Students described this 
discussion format as easy to follow and more private in the sense that they could share their thoughts in a safer 
environment. They also felt that their voice was actually heard rather than getting lost in the midst of a large group. 

Discussions around miscellaneous questions that emerged from the students’ professional practice and/or 
their needs exhibited higher levels of participation. Discussion topics were suggested by the students based on the 
needs of their current practice and in relation to the topics addressed in the courses. Students found these topics to be 
more concrete and accessible. They mentioned that when these topics arose they could thoughtfully contribute to the 
discussion as they felt well versed on the questions and showed high levels of interaction. Leveraging their areas of 
expertise and prior knowledge in these discussions was also a positive aspect reported. 

Discussion facilitated by peers as opposed to the instructor was identified as the most popular collaboration 
design. When the discussion was moderated by the instructor many students treated the discussion questions as short 
answer essay questions and not as interactive discussion. Large group discussion facilitated by the instructors was 
not found as compelling or meaningful, as everyone “answered the same thing.” The only motivation for students’ 
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participation was it being a class requirement. On the contrary, when facilitated by peers, students felt really 
connected during the discussions and motivated to participate. Peer facilitation fueled participation among students 
and created a strong sense of community. 



Conclusions 

This study reported the design and analysis of two online graduate courses for K-12 teachers. Student 
participation in a range of differently designed forums was analyzed to discover key aspects of the discourse and 
related pedagogy. 

Small group discussions that grew out of miscellaneous questions emergent from the students’ practice as 
teachers and facilitated by peers was the design of collaboration preferred among the majority of the participants. 
With peer questions, there were many questions — some with which students could not connect and others that 
worked well. Having a choice of questions to answer helped students. As one student explained: “Group 
participation was higher amongst all participants when the discussions were student led around student-needs topics 
and in a small group. We were able to talk back and forth more during the smaller discussions than during the large 
ones and we could receive more feedback from our peers.” 

Independently of discussion formats, levels of engagement decrease if the remaining coursework is not 
closely related to the discussions. If this happens, the discussions become something students would do on their 
“spare” time working on class projects. Therefore, low participation should be expected. 
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Introduction 

Trust has been identified as a benefit for team-based organizations; however it is essential to investigate 
how these benefits translate to virtual teams. This poster identifies critical elements and outcomes of trust 
building in virtual teams as well as describes relationships between them. 

The Problem 

At the university level, students should be given an opportunity to experience virtual teamwork as 
they must be prepared to communicate across cultural and organizational boundaries using technology. In the 
twenty-first century, learning while working together is becoming mandatory to meet workplace performance 
requirements, and students need to have authentic experiences in this area while earning a college degree. 
However, the challenge appears to be in designing and supporting high-performance teams as well as using 
technology effectively to facilitate teamwork. This challenge creates a major impediment for instructors to 
integrate team-based learning strategies and to continually improve team processes. The purpose of this study 
was to examine the relationship between trust and performance in virtual teams, since trust has been identified 
as the defining issue in understanding virtual teams. 

The Purpose of this Study 

The goal of this poster is to showcase a research on trust and performance in virtual teams. Because 
there is no conceptual agreement on the development of trust in virtual teams as well as its relationships with 
team performance (Langfred, 1998), this exploratory study is intended to explore this issue. 

The Proposed Framework of Trust Building in Virtual Teams 

Trust is defined in this research as “the willingness of a party to be vulnerable to the actions of 
another party based on the expectation that the other will perform a particular action important to the trustor, 
irrespective of the ability to monitor or control that other party” (Mayer, Davis & Schoorman, 1995, p.712). 
The proposed framework of trust building in virtual teams illustrates relationships between trust and team 
performance. It identifies antecedents of trust constructs as well as outcome (i.e., team performance). The five 
elements that are critical to facilitate trust building in virtual teams include: 

• Initial trustworthiness, which represents one’s readiness to have confidence in other team 
members. The trusting building process initiates with an individual determining if a person 
or organization may be worthy of trust. 



• Competence, which recognizes that all team members have the requisites, adequate 
knowledge and skills, and the qualities for the work. 

• Integrity, which consists of the firm adherence to a code of work ethics and relates to 
fairness. 

• Respect, which is linked to consideration of others, esteem, and acceptance of others’ ideas. 
It is the willingness to take seriously others’ ideas and suggestions and often is showing 
care for others. 

• Communication skills, which is the ability to share information and establish personal 
rapport with team members. It is related to the quality of how one communicates with 
others. Becoming more skillful in communication often means becoming more empathetic 
and expressing what one is saying in ways that invite cooperation, promote harmony, and 
provide constructive criticism. 

Environmental and contextual factors, described here as situational mediators, were included in the 
framework, as follows: 

• Heterogeneity of virtual teams. 

• Technology and communication level consists of the ability to access and use available 
technologies and how these technologies are used to communicate (e.g., frequency of use). 

In this study, team performance was defined as a result of teamwork products and processes and the 
relationship between the two. Teamwork processes are series of activities conducive to: 

• Efficiency processes that avoid loss or waste of energy, time, and money; 

• Effective processes that create the desired effects (e.g., good grades and good performance 
reviews); 

• Innovation of processes (application of new ideas to ensure processes’ optimization). 

Teamwork products are results of cross-border virtual teams’ processes that contribute to: 

• Team member satisfaction, that is, fulfillment of their needs or wants, professional growth, 
interpersonal relationships, sense of belonging, and pride; 

• Quality of an outcome (e.g., product, service, idea); 

• Innovation of the outcome (solution, product or intervention) to pressing challenges. 

Research Methodology 

Three courses-graduate and undergraduate level-and 18 teams of 2-5 members each participated in 
this study. These courses were offered as part of programs in Textile and Clothing and Instructional 
Technology in a large Midwestern university. The undergraduate students were mostly from the USA apart 
from 3 students from Canada, Taiwan and South Korea. Their ages ranged 18 to 29, and they were mostly 
females. Graduate students showed a dispersed age range from 24 to 48, and distributed citizenship among 
USA, Turkey, India, Ukraine and Denmark. This sample was equally distributed between male and female. 

A mixed method approach was used. Qualitative and quantitative types of data were collected to 
identify critical elements and situational mediators in the trust-building process and the role of trust in team 
performance. Face-to-face and web-based videoconferencing interviews were conducted with individual team 
members. Logs from chat and videoconference session were also analyzed. The quantitative data were 
collected over a four-month period by administering three versions of a web-based survey. The survey measured 
the constructs proposed by the framework of trust building in virtual teams. Team performance was determined 
by self-evaluation and the instructors’ grade. 



Preliminary Findings 

Be free to select any number and type of communication tools to support virtual collaboration was a 
common theme across all teams. None of them relied solely on a single medium. Instead, the teams preferred 
to use three to five technological platforms that performed diverse functions to suit various needs at different 
stages of the collaboration process. Participants believed that synchronous and asynchronous tools 
complemented each other and served different needs. 

The results indicate that the higher the levels of trust, the higher the satisfaction with the project and 
the teamwork itself. Additionally, data analysis showed that: 
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• Trust development takes time and evolves as the project progresses; some of the projects 
were not lengthy enough to facilitate this development. 

• Communication was the key to facilitate trust development. 

• Organizational climate that supports social interaction was identified as an important factor 
as well. 

Some of the factors identified as hindered trust development were: (a) delegating tasks to people who 
team members do not know well, (b) not having a formal team leader assigned to the team, (c) different styles 
of communication, and (d) different levels of commitment to the project. 

Acknowledgements: This study was partially funded by the Iowa State University College of Human Sciences 
Seed Grant Program. 
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Where’s the Oscar Award for Outstanding Online Instruction? 

Max Cropper, Joanne P. H. Bentley, Joel Gardner 
Utah State University 

The Oscar Award for Outstanding Online Instruction goes to... The most interactive course. . . NO. . . The 
course with the best instructional strategy. . . NO. . . The course with the best media. . . NO. . . . How do we 
decide? How you define “outstanding instruction” changes the outcome of any award system as much as 
adding different beads to a Kaleidoscope. This presentation will review research comparing and 
contrasting six rubrics used to evaluate online course quality. 

If there were an Oscar award for outstanding online courses, how should we determine the winner? 

Perhaps we simply should have individual awards like the Oscars have, such as best actor, best actress, best 
director, best music score, etc. For online courses we could offer awards for best instructor, best media, 
best instructional strategy, best content, best use of interactivity, best communication between students and 
instructor, best methods for collaboration between students, etc. 

However, just as the Oscar judges pick one best film that overshadows the other awards, shouldn’t we be 
able to pick one best course? If we did pick one best online course, what criteria should we use to 
determine the one best course? Should it be based on an equal balance of the other factors, or should we 
weight one factor more than the others? And what should those factors be? 

Our belief is that instructional strategy is the most important factor for a hypothetical online course quality 
Oscar award, but what is our justification? This paper reports on a three-year study focusing on online 
course quality, which has helped us identify the criteria for our mythical online course Oscar award. We 
describe the four phases of our study and the results of each. 

The thesis of the overall study was that online courses should employ effective instructional strategies, 
specifically Merrill’s First Principles of Instruction, in order to achieve the highest quality. Merrill’s first 
principles were deduced from the research literature on instructional theory, models, and standards for 
instructional design. 

Phase I 

The purpose of our study was to determine to what extent award-winning online courses, which are 
supposedly of high-quality, use Merrill’s first principles. If award-winning online courses intuitively use 
Merrill’s first principles, the construction of high quality courses would benefit from including Merrill’s 
first principles in the development process. 

In the first phase of the study, two raters (one an instructional design expert, the other not an instructional 
design expert) evaluated seven award-winning courses using Merrill’s 5 Star Instructional Design Rating 
and six other online course evaluation rubrics. 

The award-winning courses scored fairly high on both Merrill’s rating and the other rubrics. The data 
seemed to indicate that award-winning courses generally apply Merrill’s principles of instruction. 

However, interrater reliability between the raters was problematic for some courses and some rating forms, 
so the results are inconclusive. 

Phase II 

For the second phase of the study we added a baseline course which had been developed based upon 
Merrill’s first principles of instruction. We added three instructional design experts as raters, and used only 
Merrill’s 5 Star Instructional Design Rating. Phase II confirmed that most award-winning courses tend to 
implement Merrill’s First Principles of Instruction. A Bonferroni Post Hoc Comparison of Courses shows 
that there is a significant difference between the baseline NETg Excel course and the Digital Craft (mean 
difference = .63, sig. .002) and Evaluating Training Programs course ratings (mean difference = .84, sig. 
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.000). There is no significant difference between the NETg Excel course and the Research for the 
Classroom teacher, SAT, and Cashier Training courses. Figure 1 visually depicts these differences. 



Course Comparison 
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Figure 1. During Phase II of the evaluation The Research, SAT, and Cashier Training Courses scored 
effectively as high as the baseline NETg Excel course. The other two courses scored significantly lower. 
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University vs. Commercial Course Use of Principles 
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Figure 2. School courses scored high on integration possibly because instructors could assign students to do 
real-life application of learnings. Commercial courses scored low on integration because they were self- 
contained. 



Also, ratings of award-winning courses tend to be closer than we expected to ratings of the course patterned 
after Merrill’s principles. In addition, we concluded that there can be a wide variation in the 
implementation of Merrill’s individual principles. For example, school courses scored high on integration, 
while commercial courses, which are self-contained, scored low on integration. Interrater reliability was 
high except on the problem-centered principle. The problem with interrater reliability dealing with 
problem-centeredness seemed to be because of rater confusion regarding the definition of problem- 
centered. 
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How Raters Scored Courses 
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Figure 3. Comparison of raters and their ratings of course. 



Phase III 

For phase III of the study M. David Merrill, the author of First Principles of Instruction and the 5 Star 
Instructional Design Rating, served as a baseline evaluator. Having Merrill’s ratings would help us 
determine if the other raters were evaluating the same way as him. When Merrill rated the courses, he 
modified the rubric to more closely match his intent, while keeping the scoring compatible with the study 
scoring method. Merrill scored some of the courses similarly to the other raters. However, two of the 
courses which the other raters scored high, Merrill scored extremely low. See figure 3. The reason for this 
rating discrepancy was caused by Merrill’s progressively stricter definition of problem-centeredness. This 
discrepancy in scoring also suggests the need to have Merrill train the raters using his rubric, and to have 
raters reconcile ratings as they are trained on his rubric. 



Phase IV 

For phase IV of the study Merrill created a new version of his rubric and we had him conduct the training 
for his rubric. We had pairs of raters evaluate a new sample of online university courses, with each pair of 
raters using one rubric. The four rubrics used included Merrill’s Five Star rubric and the three online 
school course evaluation rubrics which we used in phase I of the study. We selected the ten courses using a 
stratified random sample based upon student ratings from existing online courses at a western university. 
This helped us identify a range of quality of courses for evaluation. We selected three low rated courses, 
four medium rated courses, and three highly rated courses for the evaluation. 

We trained each pair of raters on their respective rubric and had them evaluate the 10 courses using their 
assigned rubric. Each pair of evaluators reconciled their rating scores within one point of each other on the 
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five point rating scale for each question on their rubric. Max and Joel evaluated all of the courses with all 
three rubrics, and also reconciled their ratings within one point for each question. Max and Joel’s average 
rating for each question served as the baseline rating for each of the three online course evaluation rubrics. 

Merrill was developing his new rubric as the study was being conducted. (Show page of rubric?) When he 
presented his rubric, raters for each of the other three rubrics, who had recently read Merrill’s latest articles, 
commented that Merrill’s five star criteria were significantly more substantial than the criteria for the 
rubrics which they had been using. When Merrill presented his rubric, he hadn’t had time yet to provide a 
sample course rating. Because of the complexity of the rubric and the lack of a worked example, the raters 
for his rubric initially struggled with their rating, especially with the definition of whole tasks and task- 
centered strategy, which task-centeredness represents 30% of the total possible score. Another critical 
issue that surfaced was that some courses have a progression of tasks that lead to a whole task at the end, 
but Merrill’s rubric penalizes those courses because it requires a course to use a series of whole tasks. 

To deal with this issue, Merrill revised his rubric to provide some task-centered strategy points for a 
progression of non-whole tasks, but more points for having a progression of whole tasks. See adjusted 
portion of rubric. 

Another serious issue is that Merrill’s rubric gives extremely low scores to traditional information-based 
survey courses. Merrill’s rubric only gives credit for demonstration, application, task-centeredness, 
activation and integration. It doesn’t give points for explanation only, even if explanation is accompanied 
by illustrations. Merrill recommends traditional survey courses be converted to task-centered courses so 
that learners can apply knowledge, rather than simply memorize and forget volumes of information. 

Merrill is currently creating another refinement of his rubric that will give hope for courses that currently 
score low. After his rubric is finished and all ratings are completed, we will do a statistical analysis 
comparing Merrill’s rubric with the other rubrics. 

Conclusion 

Merrill’s five star standards are high, but they provide significant improvement over traditional instruction. 
When raters assigned to the other rubrics were introduced to Merrill’s first principles and his five star 
rubric, they concluded that Merrill’s standards were much more important for course quality than the more 
peripheral criteria measured by the other rubrics. See table 1 for a comparison of the number of rubric 
questions written about various topics by each of the rubrics used in Phase I of the study.. 

So the Oscar award for outstanding online instruction goes to. . . the course that most fully implements 
Merrill’s first principles. From the results to date we believe that Merrill’s first principles should be the 
foremost criteria for determining online course quality. However, further research needs to be done to 
determine what factors influence online course quality the most, as we study the kaleidoscope of factors 
that influence online course quality. 



95 




Task-centered Analysis 



• Is there an actual whole task? □ If not what whole task is implied? □ 

Explain: 

• Describe the actual or implied whole task. 

Description: 



Follow the flow chart to score for the task-centered principle. 

Add up to the number of points shown in the box for each question. 
The total possible points = 30. 



Score = 



Is there an authentic, real-world whole task? 



Score = 



Is there a task-centered strategy? 



Score = 



Is there a progression of whole tasks? 



Score = 

Is the whole task demonstrated? Is the 
demonstration consistent with the type of 
learning involved? (See component scoring.) 

Score = 



Are learners required to do the whole task? Is 
the application consistent with the type of 
learning involved? (See component scoring.) 

Total Score for Task-centered (0-30) = 




Figure 4. Task-Centered Analysis portion of Merrill’s latest rubric. This version provides 15 points for a 
task-centered strategy and an additional 10 points for a progression of whole tasks. If the course includes a 
whole task, it could get 30 points for this portion of the rubric. 
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Other Guidelines 
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(Merrill, 2002) 1st Principles 


Problem Centered 


Learners are shown the task, rather than just given objectives 


| Learners are engaged at problem or task level 


Learners solve progression of problems 


Activation 


Learners recall previous knowledge, experience 


Help learner see relevance and have confidence in their ability to gain 
knowledge and skill 


Use a procedure to select the right content for each learner 


| Learners given new experience for knowledge foundation [ 




Learners recall structure for organizing knowledge 








Demonstration 


Demonstrate (show examples of) what is to be learned 


Demonstration consistent with learning goal 


| Examples and non-examples for kinds -of (Concepts) \ 


| Demonstrations for how-to (procedures) \ 


Visualizations for what-happens (processes) 


| Appropriate learner guidance | 


• Learners directed to relevant information 


• Multiple representations used and compared | 


Learners assisted to relate the new information to the structure that 
was recalled or provided 


| Relevant media used 




Application 


| Practice is consistent with objectives | 


Information -about practice requires learners to recall or recognize 
information. 


Parts-of practice requires the learners to locate, name, and /or 
describe each part. 
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Table 1. Comparison of the various online course evaluation rubrics. 
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Table 1 cont. Comparison of the various online course evaluation rubrics. 



Universal Design Principles: An Overview 

Kevin L. Crow Ed.D. 

Harper College 

The term “universal design” is frequently used by educators and Web developers when discussing the 
creation of accessible electronic communication. This paper provides a general overview of the universal design 
concept. It then discusses ways in which universal design concepts can be utilized in order to help make Web-based 
electronic learning materials more accessible for individuals who have disabilities. Finally, this paper notes how 
universal design can provide added value for on-line learners who do not have disabilities. 

Universal Design 

The term “universal design” was originally coined during the 1970s by Ron Mace at North Carolina State 
University in order to describe a concept that had emerged in the field of architecture (Tobias, 2003; Vanderheiden 
& Tobias, 2006). Since that time the universal design concept has been adopted by many additional fields including 
the computer industry, telecommunications, education, and information systems (Tobias, 2003). 

Vanderheiden and Tobias (2006) claim that universal design has two major components. First, universal 
design refers to the designing of commercially available products that can be used by the greatest number of people 
without the need for assistive technologies. Second, universal design refers to the designing of products so that they 
are compatible with available assistive technologies. Vanderheiden and Tobias also claim that universal design is a 
process, not an end product; therefore, there are no universal designs or universally designed products. 

Connell et al. (1997) posit that there are seven basic principles of universal design. First, when creating a 
product, the designer should ensure that the product or design provides equitable use to all users regardless of an 
individual’s ability or disability. Second, the product or design should provide flexibility in use so that it 
accommodates a broad range of individual preferences and abilities. Third, the product or design should be simple 
and intuitive in its use. Fourth, the product or design should be capable of communicating all essential information 
under any ambient condition regardless of the user’s ability or disability. Fifth, the product or design should 
minimize any hazards or adverse consequences arising from accident misuse. Sixth, the product or design should 
cause only minimal fatigue when used properly. Seventh, the product or design should be ergo-dynamically 
appropriate for the user regardless of the user’s ability or disability. 

Tobias (2003) points out that the term “universal design” can be misleading in that it suggests that there is 
one single product that will be equally accessible to every individual. Tobias corrects this misunderstanding by 
suggesting that no one is able to create a product that will serve the needs of every imaginable user. 

Tobias (2003) notes that there are currently two general strategies regarding universal design. The first 
universal design strategy is to make a given product’s features easier to use. Tobias notes that in theory a product’s 
features should be designed so that they do not impose any burden on any user regardless of ability or disability. 

The second universal design strategy is termed by Tobias as redundancy. Tobias notes that redundancy means that a 
product will have multiple modes of interaction. According to Tobias, more interaction options offered by a product 
equals more opportunities for all users to succeed in using the product. Tobias also notes that redundancy is 
commonly built into many software products. 

Universal Design and On-line Learning 

Waddell (1999) posits that the concept of universal design pertains to computer information systems and 
telecommunications in that it calls for the development of information systems that are flexible enough that they can 
accommodate the needs of the broadest ranges of users of computers and telecommunication tools, irrespective of an 
individual’s age, ability, or disability. 

Burks and Waddell (2001) assert that there are two primary ways that the universal design process applies 
to the Internet. First, the content material should be designed in such a way that it is able to meet the needs of the 
broadest range of users regardless of age, language, or disability. Second, the Internet technology should also be 
designed so that it can be accessed by the broadest range of users regardless of age, language, or disability. 

Opitz (2002) notes that the Center for Applied Special Technology (CAST) is supporting the universal 
design process by promoting the use of versatility and flexibility within the design of on-line presentation materials. 
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Opitz notes that CAST promotes the tenet that students who have disabilities reside on a continuum of learner 
differences rather than comprising a discrete category of learner. Opitz claims that CAST promotes a universal 
design concept in which the instructor adjusts the curriculum material in order to meet the needs of all learners, not 
just those who have disabilities. 

Tobias (2003) posits an approach to the universal design of on-line learning materials that is similar in 
philosophy to Gilbert’s (1978) third leisurely theorem. Tobias purports that the origin of disability does not emanate 
from a dysfunctional or broken person; rather, the disability is in essence the gap between what a person can actually 
perform and what the environment demands. In this manner, the universal design process becomes one of removing 
the obstacles that are creating the performance gap. 

Hendricks et al. (2003) suggest that vendors of technology need to take a more proactive role in ensuring 
that universal design specifications are put into product development. Hendricks et al. claim that if more vendors 
incorporated universal design specifications into their product development, technology would be innately 
accessible, thereby reducing or eliminating the need for educators to provide additional technologies to students with 
disabilities. 

Rowland (2000) points out that postsecondary students with disabilities (on-line or face-to-face) routinely 
require computer interaction to (a) gather information about required courses, (b) resister for classes, (c) look up 
transcripts, (d) order books, (e) pay for classes, (f) utilize course management systems, (g) conduct research from 
library holdings or from the Internet, (h) get web-based information concerning campus activities, and (i) take on- 
line tests. Thompson, Burgstahler, and Comden (2003) illustrate the pervasiveness of Internet usage by universities 
by pointing out that the University of Washington had more than 200,000 web pages on its primary web server in 
the year 2002. Thompson et al. also claim that the University of Washington had several hundred thousand 
additional web pages on its ancillary web servers. 

Burgstahler (2004) offers the following seven-step process for applying the universal design process to an 
educational product. First, define the application (web site or other delivery modality) that you anticipate using for 
your product. Second, define the target audience that you anticipate will use the application. Third, identify the 
potential diversity (diversities) that may exist within the group. Fourth, apply universal design or other design 
principles to the design of the product. Fifth, apply universal design or other design principles to the design of the 
subcomponents of the given application. Sixth, develop procedures that will address the accommodation needs of 
specific individuals with disabilities for whom the standard application design does not inherently provide access. 
Seventh, test the application by using it with a diverse group of individuals who have differing abilities and 
disabilities. 

Examples of the Universal Design Process in Distance Learning 

The literature reveals several examples of the how the universal design process is being utilized to increase 
the accessibility of on-line learning, communication, and information. 

Kerscher (2004) reports that the world-wide standard known as the Digital Accessible Information System 
(DAISY) is now recognized as an ideal approach to making content accessible to all users. DAISY uses 
semantically rich extended hypertext markup language (XML) to transform textual information into a multimedia 
product that offers multisensory access to text through the use of images and sound. Kerscher notes that DAISY 
was created with universal design in mind and is currently a collaborative effort of the world-wide publishing 
industry. 

Edmunds et al. (2005) notes that the Center for Assistive Technology and Environmental Access 
(CATEA) has recently created an on-line learning object that can be utilized by all students, including those with 
disabilities. 

Curb-Cuts and other Benefits to Society 

Kinash et al. (2004) mention that disability advocates and authors often use the metaphor of “electronic 
curb-cut” to describe universal design. Curb-cuts (the portion of the sidewalk that creates a ramp from the sidewalk 
to the street) were originally put in place to be used by individuals with disabilities who used wheelchairs. 

However, after a very short time period, curb-cuts rapidly gained popularity with individuals who were nondisabled 
such as mothers with strollers, travelers with wheeled luggage, shoppers with carts, and cyclists. French and Valdes 
(2002) claim that that the universal design process helps create accessible technology that can help all students, not 
just those with physical disabilities or students who use assistive devices. 
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Burgstahler (2002b) points out that one such electronic curb-cut is the alt tag. Alt tags have been designed 
to describe images that are placed on web pages to individuals who are visually impaired by providing text 
descriptions of images. These (alternate) text descriptions of the images can be read to computer users who are 
visually impaired. Since their inception, alt tags have proven to be useful in a number of ways for computer users, 
both disabled and nondisabled. Search engines such as Alta- Vista and Google commonly locate images on the 
Internet by searching a web site for alt tags that contain a matching search term. Alt tags also permit individuals 
with slow Internet download speeds to turn off the image availability on their browser in order to maximize the 
download of content. Other examples of assistive technologies that provide benefit to nondisabled people include 
the use of real-time text captioning in video materials as a teaching aid by individuals for whom English is a second 
language (French & Valdes, 2002). Text captioning also provides benefits to individuals who are viewing television 
in a noisy environment by providing a visual-delivery vehicle for the audio content of a multimedia presentation 
(Burgstahler, 2002b). Opitz (2002) purports that the universal design process can increase the opportunity for 
learning for students who are nondisabled and have different learning styles and abilities because (in theory) the 
universal design process offers instruction in a variety of formats that can appeal to the entire learning audience. 
Calls for More Research Pertaining to Universal Design 

The literature appears to indicate that the concept of universal design is gaining ground (Moore, 2004). 
However, the universal design concept does not yet appear to be fully embraced by everyone in the educational 
community. Kinash et al. (2004) note that many current published works relating to on-line accessibility tend to 
relegate the concept of universal design to the practice of altering existing courses by “modifying the norm for the 
perceived abnormal’'’ (p. 7). Wehmeyer, Smith, Palmer, and Davis, (2004) claim that more research is needed to 
discover which universal design features provide benefit to on-line students with cognitive disabilities. Abell, 
Bauder, and Simmons (2004) claim that more research is needed in order to help advance policies and promote 
initiatives that are aimed at encouraging universal design in order to improve access to technology by individuals 
who have disabilities. Cavanaugh (2002) notes that as the population of the United States continues to get older, 
there will be an increasing need for technologies that incorporate universal design principles. 
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Teacher Education Goes Into Virtual Schooling: Developing National Models 

for Virtual Schooling Experiences 

Niki Davis, Yasemin Demiraslan, Amina Charania, Lily Compton, and Ana Correia, Iowa State 
University Center for Technology in Learning and Teaching 

Abstract 

Virtual Schooling (VS), or K-12 distance education, has become part of legislated school reform and 
improvement in many states, including Florida. Distance education is now established in K-12 schools and the 
preparation of new teachers must evolve too. Key roles in VS are: VS site facilitator, VS teacher, and VS designer. 
This paper presents a kaleidoscope of work in our national project “Teacher Education Goes Into Virtual 
Schooling,” and is particularly relevant to those in teacher education and distance learning. Descriptions of three VS 
models and real-life examples of these models are reflected in our VS lab scenarios plus their evaluation in two 
programs of preservice teacher education. 



Introduction 

Virtual Schooling (VS) is rising in both popularity and importance (Clark, 2001; Setzer, Lewis, & Green, 
2005; Zucker & Kozma, 2003; NFES, 2006; Watson, 2007; Roblyer, 2008), becoming part of legislated school 
reform and improvement in many states, including Florida. In light of the increasing demand for virtual courses and 
the rapid expansion of schools to meet the demand, it is apparent that there is a need for teachers who are prepared 
to teach at a distance from their students. Research indicates that the online instructor's role requires a paradigm shift 
in perceptions of instructional time and space, virtual management techniques, and ways of engaging students 
through virtual communications (Cyrs, 1997; Easton, 2003; Rice, 2006). 

There is a complementary need for VS Site Facilitators and other support personnel who understand the 
benefits and demands of this new mode of education and are prepared to meet its needs and requirements. Research 
in K-12 VS shows that a ‘distant’ teacher should be complemented with an adult who facilitates students learning at 
a distance (Aronson & Timms, 2003; National Education Association, n.d.; Davis & Niederhauser, 2007). 

Therefore, VS is more likely to have a beneficial impact if all K-12 teachers become competent as a VS Site 
Facilitator. 

To help meet this need, the goal of the “Teacher Education Goes into Virtual Schooling” (TEGIVS) project 
is to create a national model that integrates VS into preservice teacher education in four collaborating institutions 
(Iowa State University, University of Florida, University of Virginia, and Graceland University). VS curricula are 
being developed for three roles: VS Site Facilitator, VS Teacher, and VS Designer. This paper describes three VS 
models accompanied by examples of these models reflected in VS lab scenarios. In addition, the formative 
evaluation is also described. 



A Virtual Schooling System 

The virtual classroom includes a teacher and groups of students who are distributed among two or more 
distant schools. The teacher may also have a local class. Rather than meeting in a traditional classroom, the teacher 
and students communicate and share resources using digital technologies, such as email, videoconferencing, and/or a 
Web-based learning environment (e.g. Blackboard or Moodle). The three key roles in a VS system are VS Site 
Facilitator, VS Teacher, and VS Designer. In the typical VS course shown in figure 1, instructional designers (D) 
create instructional activities and materials with and for the teacher; the VS teachers (T) take on responsibilities 
including teaching, facilitating, monitoring, evaluating student learning, and coordinating the overall VS experience. 
Each VS Site Facilitator (F) provides immediate, personal, face-to-face communication with students, engages in 
local problem-solving of many types, and mentors students. Students (S) rely on VS Site Facilitators to provide 
information about VS possibilities, instructional support when taking VS courses, coordination of VS facilities, and 
access to VS resources. These key players are supported by administrators (A), instructional technology 
coordinators (IT) and the students’ parents or guardians (P). Of course, there is considerable overlap in these roles, 
and in some situations; individuals may take on multiple roles. Technology-mediated communication that occurs 
between and among teachers and students in the VS classroom is essential for success (Flarms. Niederhauser, Davis, 
Roblyer, & Gilbert, 2006). 
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An example of this type of virtual schooling system can be seen in some of the courses provided through 
Iowa Learning Online (ILO). ILO has produced several high quality science courses to meet demand in Iowa, where 
there is a chronic shortage of science teachers. ILO recruits and registers students, manages teachers and quality 
assurance, and supports and maintains hardware, software, and related instructional design in collaboration with 
Iowa Public Television. ILO has developed training for the ILO coaches at host schools (coach is the ILO term for 
the VS Site Facilitator). Figure 1 was developed by the project to distill the practice first developed by their lead 
mentor teacher Gail Wortmann in her teaching courses of Anatomy and Physiology, which won a WebCT award. 
This course was used in one of the TEGIVS secondary lab scenarios called ‘Nick’s Online Anatomy/Physiology 
course’ ( http ://www. public. iastate.edu/~vschooLTEGIVS/VSLab/vs%20scenarios/Scenario3/index.htmD . 



Class Offered by a VS Organization scnooi 




Do-.ignor ToucIkk F (VS S* FaaUskx) 

IT Coordinator Student Parent or guardan 

Administrator 

* ► Periodic * ^ p Technology Mediated 

Communication Communication 

Figure 1 : VS model 1 : Class offered by a VS organization to students in two regular schools 

The project developed figures for two further models in order to reflect the variety of VS experiences. The 
model in figure 2 depicts tele-collaboration that supports collaborative learning and teaching between two distant 
classes. According to Judi Harris (2001) tele-collaborative learning activities create valuable opportunities to 
expose students to multiple points of view and experiences, to communicate with a real audience using written 
language, and to expand global awareness (Harris, 2001), providing they are carefully designed. An example of this 
tele-collaboration was depicted in the TEGIVS “Teddy Bear” Scenario 

( http://www.public.iastate.edu/%7Evschool/TEGIVS/VSLab/ElementarvVersion/Scenario4/index.html ) developed 
for elementary school teachers using the International Education and Resource Network’s (iEARN) Teddy Bear 
project ( http://www.ieam.org.au/tbear/ with permission). That project stimulates on-line collaboration between 
children in different cultures though exchange of teddy bears, in order to enhance understanding and acceptance of 
diverse cultures. Children participate in the discussions as individuals or as a whole class with the guidance of their 
teachers. 

The third VS model shown in figure 3 depicts an additional section taught by one teacher in two classrooms 
connected by a videoconferencing system. An example of this model was drawn from a case study collected by two 
of the authors in Iowa. The TEGIVS secondary lab scenario ‘Danielle's Chemistry Class’ is set in this context and 
the additional web resources link with the original case study and related guidance 
( http://www.public.iastate.edu/~vschooLTEGIVS/VSLab/vs%20scenarios/Scenario2/index.html ). 
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Figure 2: VS model 2: Tele-collaborating classrooms 



Additional Section by Video Conference 

School 




School 



T (Teacher and Designer) 



Communcabon 



Student 
IT Coordnetor 

Technology Modeled 
W Communication 



F (VS Sue FeoMelor) 
Parent or guerdun 

Video Conferenca 



Figure 3: VS model 3: Teacher teaching an additional section of the class in a distant school. 

Virtual School Lab Tool Development 

The goal of the TEGIVS project is to create curriculum and related tools that are easy to adopt nationwide. 
The rapidly evolving kaleidoscope of VS described by Roblyer (2008), plus the variety of technologies and issues 
across differing content areas was a big challenge. Specifying three models helped to clarify some of the diversity. 

In addition, the TEGIVS team decided to specify the VS curriculum and then repetitively prototype both the 
curriculum and the tool interfaces in order to develop VS Lab tools that could be adapted to the variety of preservice 
programs in the US. The process began with a brainstorm session that ended with grouping the various aspects of 
VS into four main categories, namely: pedagogy, technology, assessment, and VS classroom management issues. 
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Then, the contexts were chosen for the secondary cases where multimedia was available, which were a high school 
foreign language course and two high school science courses. Fictional scenarios that challenged viewers were 
created in order to address the specific issues in using VS, including VS implementation methods, and ways to 
organize learning within the VS environment. They aimed to illustrate different aspects (pedagogy, learning 
environment, assessment, and challenges) so as to complement one another without repetition of specific items. 

In terms of pedagogy, the fictional scenarios illustrate how VS courses may be structured using different 
learning approaches including didactic inquiry and problem-based learning. The communication and interaction 
among the VS teacher, students, and content, different teaching strategies such as individual and group work, and 
variations in the flow of communication in VS courses were illustrated in designing VS curriculum. The issue of 
evaluating learning in a VS context was illustrated with several methods of assessment including reflections, 
proctored and performance-based tests, and quizzes. The three common and contrasting technologies used in VS 
courses were selected: managed learning environment (WebCT), classrooms connected via live videoconference 
(Iowa Communications Network), and a multimedia audio conferencing interface (Elluminate). Additionally, the 
scenarios presented a range of tools used to support the learning process with both synchronous and asynchronous 
modes including discussion boards, chat rooms, audio/video, email, and whiteboard. 

Following the successful piloting of the secondary lab, the elementary team teaching the Iowa State 
University course that introduces instructional technology in the early childhood elementary teacher education 
program requested a lab more suited to their students. This TEGIVS elementary lab takes a slightly different 
approach in setting the fictional scenario in a more realistic context of a newly graduated teacher in a grade 1 class 
for the teddy bear scenario and a middle school class for “Max Takes Math From Hospital.” The instructor and her 
teaching assistants were pleased with the result. 

These materials have been tested with 77 elementary and 21 secondary preservice teachers from Iowa State 
University and 133 students from the University of Florida in the fall 2006. The elementary lab has also been 
successfully tested. Results from the usability test (See Table 1 in Appendix 1) indicated that students were 
generally favorable in their ratings of the tools but also had many recommendations for how to improve their 
usefulness. The evaluation of the pilots and findings from these initial studies are described in our article in The 
Internet and Higher Education (Davis, et. al, 2007). The University of Virginia and Graceland University will adapt 
and pilot these experiences into their content-specific introductory course and methods course respectively in fall 
2007. 



Virtual Field Experience 

VS has also been introduced in an additional course at Iowa State University that prepares preservice 
teachers for field experiences in regular schools. The first pilot simply used web pages with readings and responses 
that were emailed to the instructor. Students were required to read news articles on VS and place them in the context 
of their growing understanding of the US educational system. The articles used covered the topics of VS myths, 
legislature challenges, and impact of VS on student learning and were included to stimulate more discussions and 
responses. They were also required to respond thoughtfully to questions set by the instructor, who used these during 
a debriefing seminar. This was improved in the following semester with the creation of a course in Iowa State 
University’s WebCT course management system as the tool. While maintaining focus on the content, tasks were 
organized and designed to expose students to several functions and options available in such an environment, e.g. 
online assignments, assessments, and threaded discussions. The interface improved tracking and organization. The 
new tool, WebCT, provided a secure interface for preservice teachers to post their responses, since passwords and 
user identifications were necessary. Compton, Follett and Desmiraslan (2007)conducted a preliminary analysis of 
the preservice teachers’ reactions and responses in the fall 2006 trial. Their findings also indicate the importance of 
communicating the range and diversity of VS and the need to create virtual field experiences in virtual schools. 

Conclusions and Future Opportunities 

The TEGIVS project is building VS competencies by developing tools that can be shared within the teacher 
education community. Initial findings suggest that such a tool can influence future educators’ thinking about the 
colorful kaleidoscope of teaching and learning in the 21 st century. Multiple and crisscrossed tours are necessary 
through the complex and ill-structured domain of teaching, (Spiro et al., 1988) and we are sure that this also applies 
to teaching online. 

We believe that the approach of repetitive prototyping to create materials that are easy to redevelop and 
adapt to the wide range of approaches to preservice teacher education is particularly important to the kaleidoscope of 
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VS. We aim to release all our materials under a Creative Commons license in order to facilitate further development 
both of scenarios and tools. It will be important to have scenarios that reflect pedagogic content knowledge, which 
varies with the age of the learner and their context. 

Therefore, the courses in each of the three partner universities will create further materials. In addition, the 
field and internship experiences for students will result in continual updating of technology within our program at 
Iowa State University. In fall 2007 it is planned to incorporate VS in an additional course, this time in instructional 
design to improve its applicability to the design of materials and pedagogy for VS. All these trials and pilots will 
provide us with more evidence on the effectiveness of interventions in the preparation of teachers. 

Furthermore, we are keen to hear from others who are engaged in or planning similar innovations in higher 
education. In terms of delivery, this ambitious attempt aims to influence a broader community of practice, in 
keeping with project goals. This project invites a re-examination of the complexity of both the subject of teaching 
and the context of teaching to teach. In the initial research, the medium of delivery, the curricular support, and the 
pedagogical strategies of implementation all provided both affordances and constraints to successful teaching and 
learning. If successful, the VS tools described here and those yet to be developed will reveal a new world of 
teaching and learning through the lens of VS practice. 
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Appendix 



Table 1. Overall Results of Usability Ratings Sheets on VS Tools Across Sites for 
Fall, 2006 and Spring, 2007 
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Abstract 

Adventure learning (AL) is a hybrid distance education approach that provides students with opportunities 
to explore real-world issues through authentic learning experiences within collaborative learning environments. 
Within hybrid environments, designers habitually attempt to replicate traditional classroom pedagogy resulting in 
experiences that do not support or afford meaningful collaboration and transformational learning. This paper details 
the educational, social, and technological affordances for the effective design, implementation, and research of AL 
environments, providing insights for designers and researchers of hybrid online learning. 

Ms. Anderson, a 9 th grade social studies teacher, was teaching her favorite lesson on the Louvre 
museum. One of Ms. Anderson’s most exciting days was when she took her class on a “virtual fieldtrip” to visit the 
Louvre and understand how it has progressed from a royal fortress to the museum it is today. Ms. Anderson 
developed an activity for her students to visit the Louvre website and “explore” the current and past exhibits. Her 
students progressed through the exhibits, viewed the numerous online photo galleries, and were excited to get 
started. After about ten minutes, Jenna, a student in Ms. Anderson’s class, raised her hand and asked, “What do we 
do now? Ms. Anderson replied, “This is the lesson.” 

Even with online learning growing in higher education (Lewis, Snow, Farris, & Levin, 1999) and K-12 
environments (Setzer, Lewis, & Greene, 2005; Davis & Roblyer, 2005), the levels of implementation vary greatly 
from student to student, classroom to classroom, and district to district (Setzer, Lewis, & Greene, 2005). Ms. 
Anderson’s use of an online resource is typical in the social studies classroom - online lesson enhancements that 
augment individual face-to-face lessons (Authors 1, in press). However, as Jenna’s comment reveals, students do not 
always perceive the connection to the bigger picture - the learning outcomes. They view their time on the Internet 
visiting a web site as a disparate activity from the goals of the curriculum. That is, the learning activities and 
curriculum goals do not align - an equation that does not enhance student learning. Although these disparate 
activities and types of integration are common, the movement to all-inclusive online environments (Authors 1, in 
press), where the goals of the curriculum, pedagogy, and media are in synch, is less widespread. 

An example of an all-inclusive environment is an adventure learning environment. Adventure Learning 
(AL) is a hybrid distance education approach that provides students with opportunities to explore real-world issues 
through authentic learning experiences within collaborative learning environments (Author 1, 2006). An AL 
curriculum and online environment provides collaborative community spaces where traditional hierarchical 
classroom roles are blurred and learning is transformed. AL has most recently become popular in K-12 classrooms 
nationally and internationally with millions of students participating online. However, in the literature, the term 
“adventure learning” many times gets confused with phrases such as “virtual fieldtrip” and activities where someone 
“exploring” is posting photos and text. This type of “adventure learning” is not “Adventure Learning” (AL), but 
merely a slideshow of their activities. The learning environment may not have any curricular and/or social goals, and 
if it does, the environment design many times does not support these objectives. AL, on the other hand, is designed 
so that both teachers and students understand that their online and curriculum activities are in synch and supportive 
of the curricular goals. In AL environments, there are no disparate activities as the design considers the educational, 
social, and technological affordances (Kirschner, Strijbos, Kreijns, & Beers, 2004); in other words, the artifacts of 
the learning environment encourage and support the instructional goals, social interactions, collaborative efforts, and 
ultimately learning. 

In this paper, we detail the educational, social, and technological affordances of AL environments. An 
understanding of such artifacts will enable teachers, teachers/designers, and teacher/adventurers to effectively 
design, implement, and research AL environments. Our paper follows an incremental level of complexity. We first 
examine the meaning of Adventure Learning and introduce the concept of affordances. Next, we examine the 
educational, social, and technological affordances of AL, and propose the use of established methodological 
frameworks for the effective investigation of AL environments. We conclude by looking into what the future holds 
for AL. 
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What is Adventure Learning? 

Adventure Learning (AL), a hybrid distance education approach, provides students and teachers with the 
opportunity to learn about authentic curricular content areas while interacting with adventurers, students, and 
content experts at various locations throughout the world within an online learning environment (Author 1, 2006). 
AL is grounded in two major theoretical approaches to learning - experiential and inquiry-based learning. As Kolb 
(1984) noted, in experiential learning, a learner creates meaning from direct experiences and reflections. Such is the 
goal of AL within the classroom. Additionally, AL affords learners a real-time authentic online learning experience 
concurrently as they study the AL curriculum. AL is also grounded in an inquiry-based approach to learning where 
learners are pursuing answers to questions they have posed rather than focusing on memorizing and regurgitating 
isolated, irrelevant facts. Both the curriculum and the online classroom are developed to foster students' abilities to 
inquire via “identifying and posing questions, designing and conducting investigations, analyzing data and 
evidence, using models and explanations, and communicating findings" (Keys and Bryan, 2001, p 121). Since 
Dewey (1938), numerous learning theorists have argued for the importance of providing education that involves 
students in authentic or real-world experiences in which they engage in dialogue, take action, and reflect on possible 
outcomes (Kolb 1984; Rogers 1969). The union of experiential and inquiry-based learning is the foundation of AL, 
guiding and supporting authentic learning endeavors. 

Based on these theoretical foundations, the design of the adventure learning experiences follows seven 
interdependent principles that further operationalize AL (Figure 1). 

• a researched curriculum grounded in inquiry; 

• collaboration and interaction opportunities between students, experts, peers, and content; 

• utilization of the Internet for curriculum and learning environment delivery; 

• enhancement of curriculum with media and text from the field delivered in a timely manner; 

• synched learning opportunities with the AL curriculum; 

• pedagogical guidelines of the curriculum and the online learning environment; and 

• adventure -based education. (Author 1, 2006) 



Adventure Learning Model 




Figure 1. Adventure learning model 

Some examples of AL programs are the online education programs delivered at the University of 
Minnesota since 2004. These programs include Arctic Transect 2004: An Educational Exploration of Nunavut 
(http://www.polarhusky.com/2004); and the latest circumpolar GoNorth! AL series - GoNorth!: Arctic National 
Wildlife Refuge 2006 (http://www.polarhusky.com/2006) and GoNorth!: Chukotka, Russia 2007 
( http://www.polarhuskv.com/2007 ). In all of these programs, adventurers and educators dogsled throughout the 
Arctic location of study/exploration as learners around the world collaborate and learn about the region of travel and 
the supportive content-based curriculum. Upon identifying the region of travel and the issues to be investigated, an 
inquiry-based curriculum and online learning environment is designed, developed, and delivered accordingly. For 
example, in preparing for Arctic Transect 2004 (AT2004), the development of the curriculum and online learning 
environment focused on the region of travel, the newest territory in Canada - Nunavut, and the seven Native 
communities the AL team would interact with during the six-month exploration. The curriculum consisted of ten 
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modules that were written based on three levels of curricular activities - experience, explore, and expand. 

Parallel to the development of the curriculum, the online learning environment was designed to support the 
curricular goals through the development of several online spaces. These spaces afford collaboration among 
learners, interaction with real-time authentic media from the field (i.e. the location of travel), delivery of authentic 
media that supports the curricular learning, and an overview of pedagogical principles and support for successful 
teaching of AL (Author 1, 2006). Examples of the seamless connection between the curriculum and the online 
learning environment are the online learning environment’s weekly trail updates. Every Friday during the live 
program an “education day” is taken in the field where the adventurers and educators stop traveling as the trail 
report is written and the various media that were collected during the week are downloaded, edited, and sent to 
education basecamp via satellite technologies. The basecamp manager then makes the trail report available via the 
web site by Monday at 8 AM CST. The trail report wholly supports the curricular goals. For instance, if a curricular 
unit is focusing on climate, all photos, movies, QuickTime virtual reality (QTVR) files, interviews, and trail reports, 
reinforce the climate lessons. At the same time, the education basecamp manager is updating the online learning 
environment content, scheduling the expert speaker for the week, moderating the collaboration zones where students 
from around the world are posting project files, and answering all questions from students and teachers to support 
learning and integration respectively - with all actions scaffolding the relevant curricular unit. In essence, the 
curriculum units, media, and interactions between the actors engaged in learning (i.e. learners, teachers, explorers, 
and experts) support the curricular goals of the AL environment. In the following sections, we exemplify the ways 
we view this support by providing an overview of affordances. 

Affordances: A Call for Action 

As Learning Technologists 1 , we are experiencing a tension in the field between what we understand about 
learners and how we design technology -based environments that afford learning (Gaver, 1991; Kirschner et al., 
2004). In other words, our understanding of prospective learners’ needs and abilities seldom reflects our awareness 
of the capabilities and limitations that technologies offer for instructional design. Institutions tend to develop, 
implement, and research computer supported collaborative learning (CSCL) environments and online hybrid 
learning environments with a focus on the surface-level characteristics of the pedagogical and technological 
foundations of the environment (e.g., identifying optimal group sizes, performing comparative media studies, etc.). 
These approaches often result in disappointed students and instructors, diminished motivation, wasted efforts and 
resources, and ultimately an absence of meaningful learning (Kirschner et al., 2004). What remains are merely 
“showcase environments” (p.48) that simulate traditional face-to-face communication and collaboration through 
little more than computer-assisted page turning, media galleries, and embedded chat boxes. As a result of past 
approaches, we must focus our efforts not only on the technological prerequisites for meaningful collaboration, but 
also on the educational and social conditions that fuel the nature of this interaction and experience. 

When designing an online collaborative learning environment, the selection and implementation of an 
appropriate pedagogy supportive of the instructional aims of the project, taking into account the characteristics of 
the selected media, is the primary concern (Kirschner et al., 2004). The social characteristics of the design must 
enrich the chosen pedagogy by providing engaging opportunities that encourage the social dynamics and 
collaborative interactions which exist habitually in traditional face-to-face learning (e.g., group formation, learner- 
learner and learner-instructor communication, generative problem-solving, etc.). Likewise, the technological 
foundation and design of the environment must not only allow for these social interactions to emerge, but ultimately 
thrive by providing an effective and efficient structure that satisfies users as they accomplish tasks and collaborate 
with peers in the environment. In this design scenario, Kirschner et al. (2002) refer to technology as an affordance 
for learning and education, essentially a guide for the educational and social contexts of the collaborative learning 
environment. 

Wells (2002) illustrated affordances as ecological concepts (i.e. concerned with what an environment offers 
to an unconstrained perceiver) that are relational to the user and environment (Gibson, 1979). That is, affordances 
are those artifacts of an environment that determine if and how the environment can be used by an observer 
(Kirschner et al., 2004; Norman, 1988). The archetypal example of an affordance is the door handle. Certain door 
handles are shaped in ways that lead the observer to perceive they should be pulled, rather than pushed. In terms of 
affordances, the curved C-shape of certain handles affords that the handle be pulled to open the door, whereas a 
metal plate slightly-larger than the size of a human hand leads us to believe that the plate should be pushed for a 



1 We use the designation ‘Learning Technologist’ with reference to an instructional designer focused on designing 
experiences, as opposed simply to designing instructional products or processes. Instructional designers must 
surpass the pedagogical and technical issues of developing theory-based processes and products; only then will we 
as a field design truly meaningful learning experiences (Wilson, 2005). 
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similar interaction (Kirschner et al., 2004). These relationships between the properties of an object and the 
characteristics of a user are what enable particular interactions to take place (Gibson, 1979). Though these examples 
seem fitting for the field of product design, we will discuss how affordances impact education and, more 
specifically, provide a real-world example of how they can be used to influence the design of an AL online learning 
environments. 

Although instructional designers are intent to design and develop digital learning environments in which 
the media and interactions are self-evident to learners and instructors (i.e. the design of the software makes it 
immediately clear to users how they can interact with and manipulate the environment), artifacts in the environment 
are often perceived or used quite differently than the designers original intention (Krippendorf, 1989). For example, 
recent research suggests that conversational pedagogical agents (i.e. anthropomorphic characters used for 
instructional purposes) are sometimes used by learners for entertainment purposes (e.g., casual dialogue, irrelevant 
and inappropriate questions, etc.) rather than to support learning and instruction (Authors 2, in press). The 
discrepancy between a learner’s use (or, in this case, misuse) of an artifact and the anticipated instructional 
interaction is often attributed to a weak design and implementation of appropriate educational, social, and 
technological affordances (Kirschner et al., 2004). 

The educational affordances of an online collaborative environment are those characteristics of the design 
that determine if and how learners exhibit a particular learning behavior within the given instructional context 
(Gibson, 1979; Kirschner et al., 2004; Norman, 1988). In other words, educational affordances are the properties and 
features of the environment that stimulate, engage, and maintain collaboration amongst users and encourage learners 
to interact with the instructional content in meaningful ways aligned with the chosen pedagogy. For example, when 
learners in the AL environment explore the weekly trail report (i.e. an interactive journal of photographs, movies, 
narratives, and rich descriptions from the weekly experiences on the trail during the week), they are presented with a 
number of supportive activities (e.g., collaboration zones, weekly chats, quizzes, Q&A, explorer chats, etc.) that not 
only build upon the current expedition events and topics, but also encourage learners to explore these issues in their 
local surroundings. The embedded educational affordances guide and scaffold the learner to interact with the 
environment, make use of the instructional media, and collaborate with online peers in a manner aligned with the 
AL model. 

Social affordances are defined as the characteristics of an online collaborative environment that “act as 
social-contextual facilitators relevant for the learner’s social interaction” (Kreijns, Kirschner, & Jochems, 2002, 
p. 13). Accordingly, tools and objects in digital learning environments that posses these social-contextual properties 
are called social affordance devices. Social affordances are a major facet of AL, encouraging collaboration at 
multiple levels. For example, during each week of the AL expedition, per the curriculum, students are encouraged 
to participate in collaboration zones by submitting observations and creative work (e.g., drawings, riddles, essays, 
presentations, songs, etc.) to share and discuss with other learners, teachers, and experts (both synchronously and 
asynchronously) (Author 1, 2006). These collaboration zones are social affordance devices of the AL environment 
that promote learners to engage in activities that support the social-contextual properties and goals of the AL model 
(Kirschner et al., 2004). Collaborative learning environments devoid of social affordances are “likely to isolate 
learners from their peers” (p.51), ultimately rendering the environment little more than a simple repository of 
instructional content and media. On the contrary, AL environments allow and encourage millions of students 
throughout the world to seamlessly collaborate online, an affordance that significantly impacts learning and 
motivation (Author 2, in press). 

Analogous to the social affordances of an AL environment are the technological affordances, or those 
properties of the environment that are concerned with the efficient and effective accomplishment of tasks that satisfy 
the user’s instructional intentions (Kirschner et al., 2004). Norman (2004) identifies technological affordances as the 
usability of an environment. Successful AL environments must not only be highly usable in design, but must also be 
structurally sound systems that are scalable to an influx in use. AL designers must strive to make these properties 
transparent to the users’ interactions with the environment. An online learning environment rich with educational 
and social functionalities is useless to teachers and learners if the usability aspect of the design was disregarded or 
overlooked by designers (Kirschner et al., 2004). In other words, the technological affordances of the environment 
must support the educational and social interactions. Sound usability guidelines, clear design layouts, and consistent 
navigation themes throughout an environment are a necessity as the dynamic nature and magnitude of the media 
content evolves and becomes more sophisticated over the progression of an AL program. Paired with sound 
educational and social functionalities, efficient usability and appropriate technological affordances collectively 
determine the usefulness of a hybrid distance education environment (Kirschner et al., 2004). 

The quality and effectiveness of collaborative distance education is contingent upon the “design of, and 
student’s engagement in, the learning environment” (Duffy & Kirkley, 2004, 
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p. 4). Kirschner et al. (2004) suggest that the use of appropriately designed and implemented educational, social, and 
technological affordances is the foundation for stimulating, engaging, and maintaining collaboration amongst 
learners. Accordingly, AL makes use of anchor-based, collaborative, and situated pedagogies (educational) between 
students, teachers, experts, and adventurers (social) using the Internet as a means for efficient and useful 
collaboration (technological). A shortcoming in any of these areas will result in an environment with minimal 
learning, interaction, and collaboration; in effect, a mere online journal of a person’s desire to explore the earth with 
education as the final phase of development (Author 1, 2006). It is important to note at this point that affordances 
are not simply tools or objects that can be developed as independent components for implementation into any digital 
collaborative learning environment (Kirschner et al., 2004). Rather, designers, teachers, and researchers of AL 
environments must understand and embrace the relationship between users and artifacts (i.e. devices) that exhibit the 
aforementioned educational, social, and technological characteristics. 

Affordances of Adventure Learning 

In the following sections we describe the design and implementation of three internationally-acclaimed AL 
environments - Arctic Transect 2004, GoNorth!: Arctic National Wildlife Refuge 2006, and GoNorth!: Chukotka 
2007, by providing examples and recommendations of three prerequisites for effective collaboration in AL 
environments: (a) educational, (b) social, and (c) technological affordances. 

Educational affordances of Adventure Learning 

Educational affordances are those characteristics that determine if and how effective learning takes place 
(Gibson, 1979; Kirschner et al., 2004; Norman, 1988). Within Adventure Learning (AL), these affordances are vital 
to the success of learners’ experiences becoming transformational (Author, 2006). The path to transformation begins 
with the affordance of the AL curriculum, the heart of adventure learning. The curriculum is what sets AL apart from 
an adventurer’s blog. That is, the online environment and project goals support the curricular goals. As noted earlier, 
the curriculum is written with three levels of activities - experience, explore, and expand. The words experience, 
explore, and expand ultimately coincide with the level of complexity in a particular lesson within the module. For 
example, experience activities introduce students to basic ideas or concepts. These lessons create awareness of a 
topic or issue. In some instances, students form questions that can be answered in the explore or expand lessons. 
Explore activities use experience related ideas and increase the scale in which they are viewed. Students are required 
to demonstrate an understanding of a topic as it relates to new systems and larger perspectives. An experience 
activity may introduce students to a particular plant or animal whereas an explore activity would look at population 
dynamics, predator/prey interactions, or habitat distribution within an ecosystem. Expand activities take ideas or 
concepts and relate them to new situations. Students are required to use their previous knowledge and skills to 
predict, project, manage, relate, or solve a particular question or problem. Expand activities most often involve 
inquiry-based methodology, cross-curricular research, and real-world applications. 

Each module also has two major sections - one section that focuses on the Native culture, perspective, and 
region of travel and a section that focuses on the Western perspective. This curricular design affords the opportunity 
to compare and contrast the curricular content across cultures while integrating the curriculum according to the type 
of learners. Furthermore, the curriculum is written to encourage the learner to use the online resources while also 
collaborating with peers and experts around the world. For example, as learners investigate the impact of climate 
change on Native cultures within a module, they are also encouraged to participate in the weekly expert chat with a 
climatologist from the Weather Channel™, post project files they create within the collaboration zone, ask questions 
to the adventurers/educators in the field, read the trail reports, view the media of the week, and participate in the 
online games within the online learning environment. All facets of the program are designed and developed within 
the curriculum and support each other. There are no disparate activities that do not relate to the curricular goals. 

Thus, learners are encouraged and motivated by the design of the AL program to meet these curricular goals. 

The second educational affordance, adventure-based, motivates learners and teachers to become and stay 
involved in the real-time story that is unfolding. Simply, what is “normal” and boring to one individual is many 
times unknown and motivating to another. Thus, as the team travels throughout the Arctic delivering the story, 
students and teachers have the opportunity to experience and live the story virtually. From traveling to the northern- 
most regions of Canada, Alaska, and Russia, to exploring a local town or river, the idea of an adventure motivates. 
Moreover, although it may sound simple, when the adventure involves something that everyone can relate to - dogs 
- the motivation for curricular investigation grows exponentially (Author 2, in press). Students across the world 
“adopt” their favorite sled dogs and their dog is the hook to bring them to the online learning environment in school 
and at home almost on a daily basis. Students are motivated to return to the online learning environment where they 
are going to read about the updated weekly trail report and the latest adventures of their adopted dog. For example. 
Authors 3 (2007) found that learners repeatedly returned to the AL online learning environment after school and 
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during the weekends to see “what the team and the dogs have gone through recently” and also to showcase what 
they “were working on in school” to their parents/guardians at home. 

The third educational affordance is the synched learning opportunities . All facets of the AL design have the 
curricular goals and social opportunities in mind. Within an AL program, the weekly trail updates from the field, the 
weekly media updates (e.g., photos, movies, QTVR, etc.), the local case studies, the weekly online chats, the weekly 
driving questions within the collaboration zones, and the weekly quizzes are all synched with the curriculum. 
Learners are able to receive the scaffolding and reinforcement from the design so their personal investigations into 
the curricular outcomes become transformational. Authors 3 (2007) found that students investigated a curricular goal 
(i.e. understanding the impact of climate change on native cultures) in five separate locations within the AL program 
80% or more of the time. For example, students studying climate change (1) discussed the impact of climate change 
with their teacher and fellow students, (2) posted project files that related to climate change, (3) discussed climate 
change in the weekly chats, (4) played online games related to climate change, (5) read the weekly trail report about 
climate change, and (6) watched the weekly media that consisted of interviews with Natives about climate change. 



Social affordances of Adventure Learning 

Adventure Learning social affordances are those characteristics that are instrumental in determining if and 
how social collaboration and interaction within the project take place. Within the AL model, residing next to the 
curriculum is collaboration and interaction. AL cannot be successful at a transformational level unless there is 
successful interaction and collaboration at multiple levels — between students and teachers; between students and 
subject matter experts; between teachers and subject matter experts; between students, teachers, subject matter 
experts, and the AL content; and lastly, between students themselves, teachers themselves, and between the subject 
matter experts (Figure 2). The layers of interaction and collaboration occur within the social affordance devices 
within the project. These devices include “Collaboration Zones,” “Expert Chat” zones, “Question and Answer” 
(Q&A) zones, “Ask the Team” zones, and “Send-a-Note” zones. 



Students l Students 



Students 




Experts I Experts 



Teachers I Teachers 



Figure 2. Adventure learning interaction model 

The collaboration zones, unique to each curricular unit, are socially-designed spaces within the online learning 
environment that afford learners from around the world to post and view AL project files created within the 
curriculum. For example, a learner who creates a movie for the unit on flora and fauna will upload the file to the 
“Flora and Fauna” collaboration zone. Once the movie is moderated by the basecamp manager, an interactive map 
on the front page denotes the file has been posted and the geographical location from which the post originated. 
Then, from either the Observations Map or the web page navigation, learners can view and collaborate on all the 
collaboration zone postings. Essentially, the design of each collaboration zone is specific to the curriculum unit and 
the curriculum design scaffolds learners to post their project files within this environment. 
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Although the collaboration zones are asynchronous, other features of the learning environment such as the 
expert chats are synchronous and occur multiple times throughout the week to accommodate multiple time zones. 

On a weekly basis, an expert chat is held that supports the curricular goals. For example, if the module unit is 
focused on sustainability, an expert on sustainability is asked to participate in the synchronous environment fielding 
and answering questions from students around the world. For those learners whose questions are not answered 
within the expert chats, they have the opportunity to use the Q&A zone. This zone is populated with questions that 
learners pose to the AL team throughout the program. For those questions and words of encouragement that are 
more personal, learners can ask the AL team questions or send them words of encouragement within the “Ask the 
Team” and “Send a Note” zones. 

All of these social affordance devices encourage learners to interact with the AL content and collaborate with 
AL participants around the world. The mixture of “professional and personal,” depending on the zone that is 
utilized, affords the opportunity for learners to learn more about the curricular goals while also gaining insight into 
the AL team and the daily demands of delivering an AL project from the field. This personal look into the AL team 
(people and dogs) brings the learner closer to the content and the numerous participants within the program, 
enabling learners to engage with learning experiences that are transformational (Author, 2006). 

Technological affordances of Adventure Learning 

From kindergarteners to high-school students, parents to grandparents, and student-teachers to university 
professors, the AL environments have been used by several million visitors over the past three years (Author 1, 
2006). This feat, due in large part to an expansive curriculum supported by engaging social affordance devices, was 
attainable through an efficient online design grounded in user-centered research and successful technological 
affordances. The technological affordances of an AL environment are (1) designed to ensure a highly-usable 
experience for children and adult users alike, (2) scalable to an influx of both media (e.g., trail reports, photos, 
videos, collaboration activities, etc.) and users over the course of AL project, and (3) use technology to enhance and 
guide user interactions within the environment, avoiding the use of technology for technology’s sake (Kirschner et 
al., 2004; Norman 2004). Between 2003 and 2004, the Arctic Transect environment endured a surge of visitors as 
user statistics escalated from figures in the thousands to records in the millions, with users from nearly every country 
following the expeditions and participating in the collaboration zones. Usability and scalability played a key role in 
this scenario. Flad the online environment become a cluttered depository of unorganized expedition media and 
poorly managed navigation, the environment, and more importantly the AL project as a whole, would have failed. 

Parallel to the development of usable and scalable AL online environments, it is imperative that AL 
designers select and implement technologies that support and advance the instructional aims of the project, rather 
than simply piecing together a concoction of off-the-shelf technologies that provide interactions similar to the social 
affordance devices discussed above. The selection, design, and implementation of technologies must not ignore the 
human side of the AL environment, that is, the students and teachers who will be exploring the online media and 
interacting with others in the collaboration zones (Kirschner et al., 2004). For example, the Observations Map 
(located on the overview page of each Collaboration Zone) uses technologies powered by Google Maps™ to provide 
a visual placemark that denotes the geographic origin of each interaction. The visualization technology implemented 
in the Observations Map not only provides learners with an easy-to-use reference and navigation of current 
Collaboration Zone posts, but more importantly helps learners discover and understand the foundation of authentic 
global collaboration - the collective generation of knowledge across cultural and geographic barriers. 



Adventure Learning 
Affordances 




K-1 6 students, teachers, educators, experts, parents, public audience 



Figure 3. Educational, social, and technological AL affordances 
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Adventure Learning affordances: Summary 

The thoughtful implementation of educational, social, and technological affordances in an AL online 
environment is a critical component of the AL design process (Figure 3). Engaging a wide audience of teachers and 
learners in a collaborative effort to explore an authentic context can be a complex instructional task. Thus, the use of 
sound technological affordances to mediate the social and educational interactions of users in an AL environment is 
an important framework for designers, teachers, and researchers. As more AL projects begin to surface in the 
distance education community, we encourage researchers to explore the intricate nature of these learning 
experiences through multi-methodological and multi-paradigmatic examination. The following section presents an 
overview of three such research endeavors. 

Exploring the affordances of AL environments 

Theoretical propositions regarding learning and teaching need to be empirically examined as to their 
applicability, viability, effectiveness, and efficiency. To investigate the educational, social, and technological 
affordances of AL environments, we propose the use of three established frameworks that inform each other in 
terms of the type of knowledge they generate. These three are (a) traditional performance and evaluation studies, (b) 
phenomenological investigations, and (c) design-based research explorations. In the sections that follow, we explain 
each framework with respect to AL and present an example of a research study we have conducted to illuminate the 
results that each approach may yield. It is important to note that the evaluation of the affordances of AL 
environments should not be limited by philosophical arguments of the type of knowledge generated by different 
methodological approaches. Each approach complements the other two and, in conjunction, these methodologies can 
provide a more holistic picture of AL environments with respect to variables of interest. 

Traditional performance and evaluation studies 

By traditional performance and evaluation studies we refer to research that falls under the umbrella of the 
experimental, quasi-experimental, and qualitative case study approach that examines aspects of AL in relation to 
teaching and learning. It is important to note that the label traditional should not be taken to mean that we do not 
value the importance of such research. On the contrary, such research endeavors can reveal relationships between 
variables of interest (e.g., teacher motivation, degree of AL integration, etc.), indicate new research directions, and 
inform researchers as to the feasibility of a theoretical construct (in this case AL). 

As an example. Authors 4 (2007) examined one aspect of the social affordances of AL - specifically, how 
student motivation relates to (a) student and teacher characteristics, and (b) the ways in which the AT 2004 program 
was used within the classroom. Results from a factor analysis approach indicated that students were motivated by 
interacting with the media such as photos, videos, and audio updates (social affordance devices); reading about the 
dogs, explorers’ progress, and the Inuit communities; and using the learning activities from the AT 2004 curriculum. 
Additionally, a structural equation model indicated that (a) teachers employing a traditional teaching pedagogy 
utilized AL less often than those teachers with a more constructivist teaching style, (b) AL activities significantly 
impacted student motivation, and (c) teaching style did not impact student motivation. Overall, the model suggests 
that constructivist teachers influence students’ motivation in relation to AL purely through how strongly they 
implement the AL program within their classroom. 

Phenomenological investigations 

Even though the use of the phenomenological method is not popular in educational technology circles, we 
hold that it is of utmost importance in understanding the authentic and contextual experiences of teachers, learners, 
and designers. Phenomenology is an interpretive research methodology rooted in psychological inquiry aiming to 
examine, understand, and interpret observable, yet special events in our everyday life (Heidegger, 1962). 

For example, Authors (under review) wrote a hermeneutic phenomenological manuscript describing the 
experiences of an educator/designer/adventurer when delivering AL from the Arctic. One of the constituents of this 
experience is the continuous struggle and frustrations with the technology used to deliver education from the Arctic 
to the rest of the world, a struggle to maintain the technological affordances of the environment to enhance the social 
affordances of the AL program. The adventurer notes, “ So, for three or four hours, I will be working on trying to 
get 2 Megs sent out. I’m getting frustrated. I’m getting very frustrated. I’m getting mad at the technology’. I’m 
getting really tired. It ’s now midnight. I know I have to get up the next morning to get back on the trail again. "We 
are often presented with convoluted ideals about technology: Technology is simple. Technology will make things 
better. Technology will make life better. Even though these statements may be true, they may hold accurate only in 
the environments from where they were birthed: businesses, homes, cities, coffee shops, etc. In the Arctic, 
connecting with the satellite to send a mere 2 or 3 Megs of photos “meant a day of fighting the technology to try to 
write up the report.” Technology wasn’t so simple. “We would position ourselves in a way that had a clear bearing 
to the southeast. If we had sea ice in the way, we knew we wouldn ’t be able to transmit the report. You jump back in 



117 




and fight with the server because you will connect, but it won 't transmit data. ” The adventurer endured a great deal 
every week just to shape the data into a manageable form and was rarely compensated by the acknowledgement that 
his data was actually going somewhere. It was as if he was “ throwing bottled notes into the Arctic Sea, hoping they 
would somehow find their way south around Maine, along the costal Atlantic, around Florida, and zigzag their way 
up the Mississippi river to the University to get published for the world to consume .” 

Design-Based Research 

Design-Based Research (DBR) is a relatively new research methodology that aims to assist in truly 
understanding learning in context (Brown, 1992; Collins, 1992). In short, DBR attempts to understand the “how” 
while valuing ecological validity and exploration in the messy educational contexts of the classroom and the 
distance learning environment. For example, we could ask, how do the educational and social affordances inherent 
in AL environments influence the outcomes of interest? DBR is concerned with solving real-world problems by 
interventions (Wang & Hannafin, 2005) that modify the educational, social, and technological affordances of AL 
endeavors. More formally, DBR is a multi-step methodological approach aimed at enhancing learning and teaching 
processes by means of theory development, research in authentic and naturalistic environments, and the sharing of 
knowledge amongst practitioners and researchers (The Design-Based Research Collective, 2003). Phenomena are 
studied in their “messy contexts,” outside of convoluted labs (Brown, 1992) because any insights gained from 
investigations undertaken in out-of-context environments have limited applicability in the classroom. As such, DBR 
affords us the opportunity to experiment with interventions in authentic environments to explore what happens in the 
“real world.” In line with these ideas, Collins (1992) noted that we need to methodically investigate variants of an 
intervention to accurately capture their influence. For instance, we could explore social affordances in the context of 
varying degrees of collaboration between students and teachers. Such an endeavor requires an understanding of the 
complexities of the environment in which learning occurs (e.g., for a description of school culture and its intricacies 
see Firestone and Louis, 1999), especially in the face of dominant cultural beliefs about learning and teaching that 
may prevent change (Cuban, 1993; Lortie, 1975). 

As evidenced by our proposal to investigate the affordances of AL environments with respect to varying 
and complementary research methodologies, we are in support of a multi-paradigmatic approach to research that 
may inform different facets of AL theory, programs, curricula, and learner/teacher experiences. Equally important, 
we perceive the use of the DBR framework as a valuable tool to guide us towards systematic approaches to 
designing interventions and examining ecologically valid learning and teaching processes. Finally, as DBR 
emphasizes the sharing of knowledge between researchers and practitioners, collaborating with teachers and 
immersing ourselves in contextual and authentic environments, may allow us to better comprehend what AL in the 
classroom affords. 

Exploring Adventure Learning affordances: Summary’ 

To understand phenomena of interest, researchers need to engage in systematic research endeavors. To be 
useful, such endeavors need to be multi-methodological and multi-paradigmatic, being able to inform each other in 
terms of the knowledge they generate. Additionally, such research needs to be based on solid theoretical grounds - 
nevertheless we must be prepared to amend such theories should such a change be warranted by the results of our 
research endeavors. The investigation of the educational, social and technological affordances of AL environments 
not only warrants, but demands, the use of theory-based multi-methodological and multi-paradigmatic research 
endeavors. 

Conclusion 

In this paper we discussed how the design of adventure learning addresses the educational, social, and 
technological affordances (Kirschner et al., 2004) needed for successful collaborative online learning. As the success 
stories of AL in the K-12 classroom are increasing, we can identify what is working and apply it to other online 
hybrid distance education programs while studying their effectiveness per the discussed research approaches. 
Although the design, development, and delivery of the described AL programs represents an elite approach where 
success is based on large amounts of funding, we must now use what we are learning and make it sustainable for all 
educators. The future of AL begins with educators learning to design and deliver their own AL programs while 
taking into account all AL affordances. AL does not need to be an elite form of developing learning opportunities 
where the region of travel is as remote as the Arctic. Rather, AL can be a class investigation to study an 
issue/problem within learners’ own locale using the principles and affordances of AL, leading to meaningful and 
transformational learning. 
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How can technology help bridge the cultural divide? 

A look at enterprise learning technology and its role in global business 

Kelly Dwyer 
University of Minnesota 



Introduction 

The globalization of business is an inescapable progression. In this modern technological era, global 
business is no longer limited to large multinational corporations. Thanks to the communications revolution enabled 
by the Internet, even small businesses can look beyond their local clientele to the global marketplace. As call centers 
are located across the globe from central management and production factories are situated on separate continents 
than a product’s clients, the need for intercultural communication and sensitivity has grown exponentially. 

The corporate trainer has a unique role within this expanding marketplace. Enterprise training departments 
are now required to provide training for vendors, clients, branches, factories, and other business operations across 
the globe. With the necessity of ensuring the delivery of global standards as well as preserving the unique cultural 
features of diverse populations, trainers must look to the efficiency and effectiveness of technology. The flexibility 
and ubiquity of technology provides a unique method to both ensure cultural specificity as well as control globally 
delivered content through sophisticated training systems. The goal of this paper is to investigate how enterprise 
learning technology can assist in bridging the cultural divide in the corporate training environment. 

Globalization and the Cultural Divide 

Several factors have accelerated globalization in the business world: from international trade agreements to 
reduction of overhead through outsourcing; from instantaneous global communication and financial networks to the 
increase of immigration (Thomas & Inkson, 2004, p. 7). Global business is growing. “Globalization means an 
increase in the permeability of traditional boundaries, including those around countries, economies, industries and 
organizations” (Thomas & Inkson, 2004, p. 6). Although boundaries of time and space can be seamlessly ignored in 
global business, intercultural communication has the potential to make or break businesses. “Unlike legal, political, 
or economic aspects of the business environment, which are observable, culture is largely invisible. Therefore, it is 
the aspect of global business that is the most overlooked” (Thomas & Inkson, 2004, p. 8). 

What is the cultural divide? 

While defining ‘culture’ can be difficult, the cultural divide is a simpler concept. Considering the 
complexity of what constitutes culture, a broad but necessary definition is “the way of life for an entire society” (D. 
Jary & Jary, 1991, p. 101). This can include, but is not limited to, language, dialects, patterns of speech, food, sports, 
values, clothing, concepts of authority, art, social hierarchies, religion, mannerisms, beliefs and politics. Culture is a 
broad sweeping idea that can represent any unique element that defines a society. It is a distinguishing measure to 
separate one society from another. 

Culture can be implicit and explicit as seen in the metaphor as culture as an onion (Spencer-Oatey, n.d., p. 
3-4). In descending order from explicit elements to implicit elements, culture can be defined as artifacts & products, 
rituals & behaviour, systems & institutions, beliefs, attitudes & conventions, and basic assumptions and values 
(Spencer-Oatey, n.d., p. 4). “Culture shapes the cognitive schemas which ascribe meaning and values to 
motivational variables and guide our choices, commitments, and standards of behavior” (Erez, 1994, p. 13). Culture 
is a part of who we are and is evident in every social interaction, whether within or outside of our own societies. 

The cultural divide refers to the meeting of different cultural elements and the differences between them. 
How well or poorly one is able to understand and accept the differences between one’s own culture and others 
defines how well one can bridge the cultural divide. Regardless of how well versed or educated a person is in 
another culture, miscommunications, misunderstandings and blunders are a commonality. 
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Why do we need to bridge the divide? 

From call centers to industrial manufacturing, from medical equipment production to technology design 
centers, corporate training departments now have to deliver core business competencies and values to people 
anywhere in the world. Considering cultural diversity in the development and delivery of training is not only an 
ethical requirement of the job, but also a professional necessity. Fostering and valuing a multicultural community is 
one of the key ethical values cited by both the American Educational Research Association (AERA) and the 
Association for Educational Communications and Technology (AECT). The AERA code of ethics states that 
“[rjesearchers have a responsibility to be mindful of cultural, religious, gender, and other significant differences 
within the research population in the planning, conduct, and reporting of their research” (AERA, n.d.) and the AECT 
code of ethics states that its members “[sjhall seek to encourage the development of programs and media that 
emphasize the diversity of our society as a multi-cultural community” (AECT, n.d.). 

Cultural barriers can have a significant impact not only on how training is received and validated by a 
learner, but on the overall reputation and future productivity of a corporate training department. It is common 
business sense to understand that business success in any culture is dependent upon variables such as trust, respect 
and confidence. These concepts are cultural constructs. In order to be successful, a business must communicate these 
values to a client in a manner that ensures cultural understanding and acceptance. Understanding and 
communicating with other cultures is as important in business as understanding other languages. 

Bridging the cultural divide in corporate training can be as basic as ensuring that training is communicating 
the appropriate message and information to the learner. Mistranslation of content can cause significant 
misunderstandings due to either inappropriate word choice or diffusion of the overall message of the content. If 
learning technology is not culturally flexible, there can be large consequences for the success of a training program: 
training dates may be incorrectly stated to a learner, times of sessions may be confused, etc. Inappropriate content, 
confusing design and inaccessible materials can cause a learner to loose trust in a product. And once trust is lost, it is 
difficult to regain (Karvonen, Cardholm, & Karlsson, 2000, p. 4). Ensuring the success of a delivered training 
program to multiple cultures is far more complex than simply translating content. 

In 2002, Thomas, Mitchell and Joseph proposed altering the standard training development cycle, ADDIE 
(i.e. Analysis, Design, Development, Implementation, Evaluation), to include a cultural dimension (Sieffert, 2006, p. 
9). This ‘third dimension’ of ADDIE would include the values of intention, interaction and introspection. 
Instructional designers must approach their projects “in a manner that is culturally sensitive and grounded in the 
notion that culture is inescapable” (Sieffert, 2006, p. 9). Once the intention of cultural sensitivity is grasped, Thomas, 
Mitchell and Joseph propose that instructional designers must interact with the target audience of the product to 
ensure cultural appropriateness. And lastly, “[djesigners must examine their own values and beliefs to ensure that 
they are not unintentionally including them in their design (Sieffert, 2006, p. 9). 

Enterprise Training Technology and Culture 

When training managers are located continents away from where they are responsible for delivering 
training, how can they manage the cultural requirements of doing business overseas effectively and efficiently? In 
comparison with other professions using instructional technology (e.g. K-12 and higher education), enterprise 
training departments most often utilize advanced technology to manage and deliver their training. Learning 
management systems (LMS’s) and eLearning are technologies used extensively in enteiprise learning. These 
technologies have the potential to bridge the cultural divide for enterprise training departments. This section 
identifies elements of LMS and eLearning technology that are at risk for being culturally exclusive. These lists are in 
no way comprehensive. Many other elements of enterprise learning technology can impact cultural acceptance of 
learning. These lists were complied to serve as an introduction to the types of elements that can be used to bridge the 
cultural divide. 

Key Technology 1: Learning Management Systems 

Learning Management Systems are software solutions used for the administration, delivery and reporting of learning 
activities. LMS’s are used by a wide variety of organizations such as higher education, k-12 education, small and 
large private businesses in all industries and the government. These organizations use LMS’s to deliver eLearning, 
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manage the enrollment and training delivery process, administer and assign training, and track and report training 
delivered not only through the LMS, but also face-to-face and blended learning solutions. LMS’s are highly flexible 
in design and functionality and can include features such as assessment and testing, evaluation, certification 
management, ERP/CRM integration, learner and manager self-service, knowledge management, competency 
management, performance management, content authoring and logistics management. 

Learning Management Systems have many features that are specific to the culture in which they were 
developed. Considering that the business of enterprise learning revolves around concepts such as scheduling, content 
delivery and communication, culture permeates the very design of these technologies. However, LMS’s can be 
designed or configured to take intercultural differences into account. A review of key considerations for cultural 
specificity is provided below: 

Table 1: Cultural Specificity in Learning Management Systems 



Salutation 


The very first thing a learner sees when they log into a LMS is a greeting with their 
name, informing them that the systems recognizes them as a learner and will be 
tracking their learning progress correspondingly. However, name construction and 
configuration is very different around the globe. Many cultures order the family 
name before the individual name, and the terms “first name” and “last name” may 
not be recognized. LMS’s should be able to display a preferred name or nickname 
since some Asian cultures also have a westernized individual name. The system 
should be able to clearly differentiate between multiple names, as are found 
commonly in Spanish family names. Integration with the eLearning standard, 
SCORM, can also be culturally dependent since the SCORM standard requires the 
transfer and communication of a learner’s full name, which can conflict with the 
cultural use of single or multiple names. 


Language 


Different languages have unique formatting requirements. Do not forget that many 
languages are read from right to left, while other languages are read left to right. This 
reading order can impact not only the screen design and layout of a LMS interface, 
but also how intuitive it is for each learner to determine where to go to complete a 
task within the interface. Automatic language detection is a critical part of ensuring 
cultural flexibility. 


Course Catalog 


Course offerings should be displayed to the learner within a LMS dependent upon 
their location, language and business area. Filtering course catalogs can be difficult to 
implement due to different preferred languages within a region. The commonality of 
expatriates causes difficulties for LMS’s because they require constant integration 
with HR systems to determine the accurate location and language preferences for 
employees on short-term and long-term assignments. 


Scheduling 


A main function of a LMS is to manage scheduling of synchronous training. The 
LMS must ensure that time and dates are displayed in the appropriate formats, such 
as MM/DD/YYYY for the United States. ISO Format of YYYY-MM-DD may not be 
recognized by some cultures. Times are also displayed in either 24-hour format or 12 
hour format with AM/PM designators. Time zones are very difficult elements to 
integrate into scheduling due to the differences with daylight savings shifts across the 
world and countries or states where daylight savings is not taken into consideration. 


Communications 


A critical feature of a LMS is its communication systems. Emails can be sent to a 
learner reminding them of an upcoming training course or deadline, inviting them to 
participate in a course, reporting on their completion, communicating pre-course 
materials, etc. The first aspect of LMS communication is the author. It is important to 
consider whom an email comes from in context of it’s content. If an email is 
demanding a completion of overdue required work, then an email from a 
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disembodied global address is less like to be given as much attention as one sent 
from a global head of training, or the head of one’s own division. Authority is a 
cultural aspect that relates to concepts of power distance between subordinates and 
authority figure (Thomas & Inkson, 2004, p. 34). Ensuring that the appropriate 
authority communicates important messages will greatly impact the way the 
messages are received. Emails should be sent in the appropriate languages that 
learners require. While some email systems are incompatible with formatting and 
styles, be aware of cultural meaning in color (see eLearning section below). The 
content defined for each LMS communication email template should be flexible in 
order to be customized to take advantage of cultural differences in communication. 
For instance, American workers require clear and precise communication of 
deadlines and requirements. However, this approach would be considered blunt and 
improper in other cultures. The difference between inviting and requiring attendance 
is an enormous cultural boundary that can be managed through the words and content 
specified in communication from a LMS. 


Grading 


Grading and assessing one’s performance on a learning activity is considered a 
required measure of completion in many cultures. However, indicating that a learner 
may have performed poorly or failed a particular course or test is prohibited in many 
cultures. For instance, the German Worker’s Council prohibits LMS’s that are used 
by Germans to include any status other than “completed” or “in progress.” A LMS 
must be flexible in its technology in order to meet such standards of terminology. 


Evaluation 


Course and instructor evaluations are an important part of the instructional design 
process. Constantly improving content requires the honest feedback of participants. 
However, many cultures do not prefer to provide critical feedback. It is not culturally 
acceptable to challenge an authority figure (Thomas & Inkson, 2004, p. 34) in many 
Asian and South American cultures. Instructional designers need to be careful with 
the wording and features such as anonymity when creating and delivering 
evaluations. 


Deadlines 


Certain cultures have high values of uncertainty avoidance, which can affect the use 
of LMS technology (Thomas & Inkson, 2004, p. 34). Some cultures may require 
structural elements such as deadlines in order to avoid confusion over expectations 
for completion, regardless of whether the training is mandatory or voluntary. 
However, some cultures will react with a natural resistance to deadlines, such as 
American cultures, where imposed structural constraints are seen as a barrier to 
productivity. 


Help options 


Different cultures have different expectations about the level of assistance required 
from owners of enterprise technology. Also, many cultures have varying levels of 
technical proficiency. As a result, enterprise training departments must implement the 
most effective help desk solutions to support their training products and learners. 
Factors to consider should include locally situated or globally centralized help 
centers, language support for call centers, multi-language help guides and 
walkthroughs, and local or global phone numbers for support. Each of these choices 
will impact the perception of a corporate training department within each culture. 
Some cultures require hands-on supervision training and support, where others are 
reticent to contact anyone for assistance. Regardless of culture, help centers must be 
well staffed and well trained. Universally, competence is a primary component to 
gaining and maintaining trust. 
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Key Technology 2: eLearning Courseware 



The delivery of eLearning is a popular and cost-effective method for ensuring content delivery around the 
globe. However, as instructors normally take into account the audience of a course to ensure it’s effectiveness, so 
must instructional designers when designing an eLearning course. The following elements of eLearning must be 
considered in order to avoid cultural miscommunications and misunderstandings. 

Table 2: Cultural Specificity in eLearning Courseware 



Color 


Color has many different meanings for different cultures. For instance, red in China 
indicates prosperity and rebirth, but to Americans, red means stop or danger. To 
predominantly Christian cultures, white represents innocence and purity, but to the 
Japanese, white is the color of death and mourning (Hackett, n.d., p. 7-8). Before 
using colors, an instructional designer should research their cultural values to avoid 
incorrectly communicating the message of the content. 


Sound 


It is important to consider not only language, but vocal intonation and accents when 
providing narration in an eLearning course. Providing narration in English spoken 
by an Australian to an American audience can be confusing and can lead to poor 
consideration of the program. Also, sound effects may not be universal in every 
culture. Everyday items such as alarm clocks or ambulance sirens sound different 
around the world. 


Images 


Images are some of the most culturally exclusive elements to be included in 
eLearning training. Instructional designers should be aware of the cultural values 
associated with images of people. Clothing, race, facial expressions, physical 
contact, the role of women in the workplace, eye contact and hand gestures are all 
factors that can be grossly misinterpreted in different cultures. For instance, an 
image of a female boss in a modern business suit and skirt, shaking hands and 
making eye contact with a male subordinate worker would be insulting in many 
Arabian cultures. Symbols that represent values such as currency, everyday items 
and concepts can also be culturally specific, such as the shape of an electrical plug 
or a stamp for mailing letters. 


Navigation 


Learners from high uncertainty avoidance cultures may prefer to be guided through 
eLearning courses in a clear and linear style (Thomas & Inkson, 2004, p. 34). 
However, other cultures may react to guided and linear navigation with boredom or 
frustration. 


Language 


Although translation may be one of the most obvious methods of bridging the 
cultural divide, it is also a very difficult process. Translation is not only time 
consuming and expensive, but, depending on the content, mistranslation can be 
deadly. For example, a pharmaceutical company’s training department translates 
their standard procedures for production into other languages. If simple content such 
as safety procedures are inappropriately translated, the company not only could face 
significant loss of reputation and business, but lives may also be lost. Beyond literal 
mistranslations, instructional designers can have a difficult time ensuring that 
inference and abstract messages are preserved across translations. It is important to 
ensure that content is translated by a native speaker in the appropriate dialect for the 
target culture. When writing for translation, ensure that content is formatted simply 
and avoid over styling text. Do not include metaphors, jokes, or slang unless the 
translator is able to substitute an appropriate cultural equivalent. Be aware of the 
way that differences in text directionality will cause learners to navigate through the 
course. The number of words in a sentence and the number of letters in a word are 
enormously different across languages and can impact how much text can appear on 
a screen. Be aware of the unique characteristics of the languages that the content 
will be translated into and design appropriately. 


Laws & 

Authority 


Every culture has different concepts of authority and different laws that instructional 
designers should be familiar with. What constitutes sexual harassment in France is 
not the same as in the United States. All instructional content should be sensitive to 
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the laws and regulations of other cultures. Portrayal of authority figures should be 
culturally explicit to the target audience. For instance, a subordinate providing 
honest feedback or complaining to a boss is not an appropriate portrayal of the 
interaction with authority figures in Asian cultures, while it is the norm for 
American and many European cultures. 


Text specifics 


There are a large number of inherent additions to content that should be considered 
when developing multicultural content. These include but are not limited to units of 
measure, punctuation, number formatting, list separators, acronyms, abbreviations, 
personal titles, symbols, time zones, and date formats. If materials are available to 
be printed from eLearning courses, ensure that the formatting of the materials is 
compatible with all standard paper sizes, including A4 in European countries and 
Letter paper in America. 


Pedagogy 


Individualist and collective cultures will approach the use of eLearning in different 
ways and instructional designers need to be aware of the pedagogical impact of 
these differences. Collectivist cultures are more comfortable when relating their 
actions to a larger group where as in individualist cultures, the consequences of an 
action are considered only in relation to the self, not the group (Thomas & Inkson, 
2004, p. 30). This can impact the structure of the eLearning product (blended with a 
synchronous group interaction component, or guided individually-paced learning). 
Each culture has developed a unique method of instruction and learning and many 
of these methods may be incompatible. For instance, constructing an eLearning 
scenario as a game could be considered frivolous and insulting in countries that do 
not value play as a part of learning. Matching cultures with pedagogical styles will 
ensure that learners not only accept the training, but have a better opportunity to 
internalize and transfer the knowledge. 



It is important to be aware of the danger of surface frivolity. Including cultural elements that are unrelated 
to the content of the course could be perceived as insulting and flippant. Bridging the cultural divide is not as simple 
as replacing images with people of the correct races or ensuring that translated text can fit on a screen. It is very 
important to ensure that content, whether globalized or localized, is respectful of its audience. 

Globalization versus Localization 

Globalization experts identify a significant rift between global standardization and the need for localization. 
Localized products are customized to include culturally specific features, whereas globalized products have had all 
cultural specificity stripped away to produce globally acceptable content (Hackett, n.d., p. 2). Internationalization is 
the act of preparing a product to be localized and involves identifying all aspects of the product that are culturally 
specific in order to easily substitute local values (Hackett, n.d., p. 2). 

The implications of the debate range from the negative impact that globalization can have on acceptance of 
products, to the overwhelming effort and time investment that internationalization and localization requires. Some 
state that globalization can provide a neutral and safe approach to delivering content to multicultural audiences. 
However, as demonstrated above, the need for cultural specificity should far outweigh the need to save the time and 
effort required to produce a culturally specific product. By sacrificing the unique cultural characteristics required to 
create a meaningful learning product for different cultures, an enterprise training department could be at significant 
risk of failure. Within the training development cycle for each learning product, instructional designers should 
follow Thomas, Mitchell and Joseph’s suggestions and determine the best way for each culture to fulfill the defined 
learning outcomes of the product. Advocating localization will ensure the preservation of trust and attention of the 
enteiprise learner. 
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Concluding Remarks 



Instructional Designers and Technologists need to be aware that the technology that they develop to 
manage and deliver intercultural training will undoubtedly be based on their own personal cultural assumptions. It is 
critically important to be aware of the cultural divide and to develop and utilize technology with a strong cultural 
intelligence of one’s audience. “Cultural intelligence involves understanding the fundamentals of intercultural 
interaction, developing a mindful approach to intercultural interactions, and finally building adaptive skills and a 
repertoire of behaviors so that one is effective in different intercultural situations” (Thomas & Inkson, 2004, p. 17). 

A checklist of questions should be answered prior to beginning any project, for example: What are the 
cultural backgrounds of the audience for this project? Are there culturally unique or sensitive topics that will be 
covered in this project? Will the project be globalized for consistency or localized for specific cultural requirements? 
What appropriate pedagogical approaches are compatible with the target audience? Will the project need to be 
translated, and into what languages? Just as a proofreader approves content, a cultural proofreader should approve 
projects for multicultural audiences. Ensuring that content is culturally appropriate and well received will ensure 
success of your enteiprise learning project. 

Considering cultural aspects of technology in the workplace is the first step towards implementing a 
culturally intelligent training program. Enterprise training departments must ensure that cultural elements are 
included in every area of project management, design and delivery. 
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Introduction 

Competency-based education (CBE) has emerged as a powerful variation of distance higher education. As the 
technology of the Internet has catapulted distance learning as a widely accepted alternative to classroom-based 
campus education, several institutions and many programs have incorporated CBE to further enhance the adult 
learning experience. Distance learning enables access, as working adults can upgrade their skills and prepare for 
new careers through the convenience of working on their own time schedule from home or office computers. CBE 
enables adults who have already acquired knowledge, skills, and abilities from prior college attendance or through 
work and community experience, to incorporate those substantively into degree programs. 

What is competency-based education? Its major premise is that diplomas and credentials should be awarded on the 
basis of demonstrated student performance - on outcomes and results - rather than on the accumulation of credits, 
the number of successful semesters completed (seat-time), and adherence to campus residency requirements. CBE 
is often tied directly to the occupational requirements of working professionals, requiring degree candidates to solve 
real-world problems of nurses, teachers, computer scientists, and business managers. CBE espouses liberal 
education competencies as it challenges students to effectively apply skills like critical reasoning, quantitative 
literacy, language, communication, the humanities, and the natural and social sciences to adult roles of citizen in a 
participatory democracy. CBE often defines outcomes around the standards that have already been set and are 
continually redefined by state and national government agencies, professional associations, and accrediting bodies as 
levels of knowledge, skill, abilities, and dispositions to which graduates must achieve. Not only does a CBE 
credential explain the students’ capability to demonstrate proficiency in required areas, CBE is also attractive to 
employers who feel certain that CBE graduates possess the abilities to “hit the ground running” in the positions to 
which they are recruited or promoted. 



Although virtual institutions, like Western Governors University (http://www.wgu.edu), have embraced CBE as 
their defining characteristic, it is more typical that CBE will be adopted by professional programs within institutions. 
SUNY Empire State College’s competency-based MBA program is a good example of this where one degree 
program adopted CBE within the SUNY University System ( http://www.esc.edu ). DePaul University’s School for 
New Learning ( http://snl.depaul.edu ), began one of the first CBE undergraduate residential programs thirty years 
ago, and offers both undergraduate and graduate hybrid, and distance education to local, regional and international 
constituencies. Degree programs based on CBE have emerged in such areas as nursing, business, information 
technology, education, dentistry, and engineering. Accrediting and professional associations are becoming much 
more concerned with the outcomes of higher education, not just the inputs of what students are taught. Higher 
education institutions, such as Excelsior College (https://www.excelsior.edu/) and Thomas Edison State College 
( http://www.tesc.edu/ ) don’t consider themselves competency-based, yet they have high-stakes assessments where 
candidates demonstrate their abilities in proctored settings, leading to credits accepted at hundreds of other 
institutions. Similarly, credit-by-evaluation practices involving portfolio development with work samples are 
practiced at numerous postsecondary institutions and overseen by the Council for Adult and Experiential Learning 
(CAEL) which concerns itself with candidates’ mastery of competence in their award of credit 
( http://www.cael.org/ ). Many CBE degrees have site-based requirements, such as the taking of assessments at 
proctored sites, classroom observations of candidates’ teaching abilities, and residencies; however, many CBE 
programs, such as DePaul’s and WGU’s, are delivered either primarily or completely online. 

Interestingly, competency -based education is not just an American phenomenon. Higher education institutions 
throughout the European Union are incorporating CBE within their degree programs as they comply with the central 
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elements of the European Higher Education Area mandates: to base credits upon student workload, to have 
comparable degree programs, and to promote mobility and transfer from one institution to another, regardless of 
country (Colledo, 2007). Institutions such as the Spanish National University of Distance Education (UNED) are 
restructuring degree programs to incorporate assessment for prior learning and redefining courses to include 
required competencies (Santamaria, 2007). 



Rationale for CBE 

The rationale for CBE dates back to ancient apprentice programs where young workers lucky enough to be chosen 
would work for several years under the tutelage of a master craftsman until proficient in their professions. 
Educational programs also evolved from tutelage of master teachers to formal educational programs, yet there was 
often a separation of formal education from mastery of career competencies. “The once clearly marked distinction 
between teaching and training has become an increasingly permeable boundary” (Taylor, Marieneau, & Fiddler, 
2000, p.xi). As careers and education have intertwined more over the last century, educational programs have begun 
integrating competency-based courses that assess proficiency of specific subjects for mastery of knowledge in those 
areas. “Starting shortly after the end of World War I, there began emerging. . .a growing body of notions about the 
unique characteristics of adults as learners. . ..these notions evolved into a comprehensive theory of adult learning.” 
(Knowles, 1984, p.28). 

In adult education the curriculum is built around the student’s needs and interests. Every adult person finds 
himself in specific situations with respect to his work, his recreation, his family-life, his community-life, et 
cetera — situations with call for adjustments. Adult education begins at this point. Subject matter is brought 
into the situation, is put to work, when needed. Texts and teachers play a new and secondary role in this 
type of education; they must give way to the primary importance of the learners. (Lindeman, 1926; as cited 
in Knowles, 1984, p.29). 

CBE has gained appeal for working adults because it acknowledges their life experiences and allows them 
opportunities to apply those experiences for college credit. This is particularly meaningful for adults who may not 
have been able to attend or complete college because they can now use their life experiences to demonstrate mastery 
in their fields of expertise. Adult learners are goal-oriented, they appreciate that CBE presents educational degree 
programs in a clearly-defined and organized context. Adult learners are also practical; they want to feel that 
returning to college is meaningful to them. Working on a degree after a number of life successes has to make sense, 
either for furthering their career opportunities, or for a sense of personal achievement. But adult learners often feel 
squeamish about embarking on a degree pursuit because their earlier college experiences may have been less than 
positive. They may need affirming reinforcement and ongoing interaction to bolster self esteem when returning to 
college. CBE helps students become more confident and proficient through engaging them and providing 
customized programs that take into account individual differences in competencies, in the pace of learning, and in 
individual learning styles. “The learning that adults do arises from the context of their lives, which is in turn 
intimately tied to the sociocultural setting in which they live. . .thus learning in adulthood is characterized by its 
usefulness for immediate application to the duties and responsibilities inherent in the adult roles of worker, spouse, 
parent, citizen, and so on.” (Merriam & Cafferella, 1991, p.304). “If the meaning of an experience is not 
immediately evident, there is an inclination to give up and move quickly on to something else” (Taylor, Marieneau, 
& Fiddler, 2000, p.25). What can be so effective in CBE programs are the opportunities to take the adult learners’ 
positive experiences and help facilitate students’ transformation to mastery of learning through mentoring, dialog, 
positive reinforcement, provision of new opportunities for learning, and reflection. 

Most CBE programs have stringent assessments that require students demonstrate their proficiency in the desired 
areas. Once they acquire each competency credit or unit, students continue to build on their 

successes towards their degree. CBE provides a safe environment where learners can scaffold new learning on their 
prior experiences and presents them with new educational choices and alternatives. Adults want to have the 
opportunity to achieve their goals, and CBE helps them by matching their level of development and self-motivation 
with their desired degree goals, and then provides them with an environment that nurtures and facilitates the 
attainment of those goals. 

There has been a burgeoning of distance degree programs over the last decade because online instructional delivery 
allows students to study conveniently at the time and place of their choice. In addition, CBE programs also present 
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the advantage of accelerating through a degree program for those students who may already have had a number of 
life experiences that can translate into competency mastery and who are willing to study to acquire new ones as 
quickly as their pace allows. 



CBE Practices 

All institutions of higher education must address similar issues, such as what degree and credential programs to 
offer, how to attract students that are a good fit form the institution, how to build upon what students already know 
and can do, how to assist students in planning their degree program and progressing through it, how to maximize 
their learning experience, and how to determine whether candidates have mastered the material to the level required 
of a graduate. In the section below we describe CBE practices in program development, marketing and recruitment, 
admissions, prior learning assessment, advising and academic planning, learning, and outcome assessment as they 
are shaped by various institutional practices. Throughout this explanation, the authors used our own institutions, 
Western Governors University and DePaul University’s School for New Learning, in describing practices, as well as 
bringing in examples from other institutions as appropriate. 

Program Development 

Competency-based institutions must determine which degree programs to offer, and all CBE programs must 
determine which competencies should make up their degrees, domains, courses, and so forth. The usual practice is to 
develop internal standards, relying on external ones where available. DePaul University’s School for New Learning 
(SNL) uses internal academic staff and international faculty to develop program competencies based upon state, 
professional association and regional accreditation standards. It also maintains a faculty expert in adult learning. 
Western Governors University relies upon its senior faculty embodied in its Program Councils (PC) to determine 
degree programs, domain areas, and competency outcomes. This PC faculty is comprised of experts from academia, 
business, industry, foundations, and non-profit companies in areas where WGU has developed programs in 
education, business, information technology, and health professions. Additionally, WGU, SNL, and other 
competency-based program generally use the external standards of competency established by state education 
offices, professional accrediting bodies, and industry certifications as means of adopting and customizing its 
competencies. 

Marketing and Recruitment 

Those individuals who already possess competency through prior work and academic experience are those most 
attracted to competency-based education. DePaul’s SNL relies almost exclusively on its reputation as the largest 
Catholic University in the United States and on word-of-mouth to promote its competency-based programs. Western 
Governors University does little mass-marketing; instead, it works with several Internet-based lead-generating 
enteiprises to reach out to interested adult learners. Word of mouth from current students and successful graduatess 
becomes increasingly important to reach prospective students as the University flourishes. WGU employs a cadre of 
enrollment counselors to communicate by e-mail, online chat, and telephone with prospective students, explaining 
the CBE model of distance education, the rigorous demands of online study, and the support structure of the 
University to enable CBE online. WGU, as a private non-profit institution, is particularly interested in recruiting 
persons currently underrepresented in higher education: those with rural residences, from ethnic minorities, from 
lower income households, and individuals who are the first of their families to attend college in their generation. 
Currently 83 percent of their students come from one of more of these categories (Eastmond, 2007). SNL’s resident 
degree program began 30 years ago in inner-city Chicago, and has always served a high proportion of underserved 
students. It now extends those opportunities to distance and remote students all over the world through its online 
degree programs. 

Admissions 

It may not be true of all CBE offerings, but generally these programs and institutions have a more open approach to 
admission. This is both is terms of the frequency of admissions and the requirements for attendance. Prospective 
students at DePaul’s SNL work first with an advisor to take an academic assessment. Afterwards they attend a 
learning assessment seminar that appraises their abilities to self-regulate, to think critically, and to communicate 
effectively in writing so that they can be assisted appropriately to succeed in the program. Western Governors 
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University admits students monthly after they have successfully completed an admissions assessment that screens 
applicants who will succeed in online CBE from those who will not. The assessment includes such areas as prior 
education (including required degrees - high school for undergraduates, bachelors for graduate students), math and 
language abilities, technology skills, students’ context for support, and their commitment to weekly academic 
workload. Upon admission, WGU also requires all students to take an initial course (self-paced, lasting roughly two 
weeks) that orients them to online study, acquaints them with the library, and introduces them to their mentor. As a 
final exercise the student and mentor together they plan out an individualized academic action plan. This plan 
contains the sequence of assessments and associated learning resources, along with dates for completion for the 
degree. 

Prior Learning 

Universities and colleges typically affirm students’ prior college-level learning in one of two ways: awarding credit- 
by-evaluation, and through assessment. WGU uses the assessment, whereas SNL uses both. In the first approach, 
students produce portfolio of relevant materials that document their competencies during the credit by evaluation 
process, usually over a period of several months as part of an initial course and under the guidance of an academic 
advisor. They document credit earned through prior college work, they investigate credit recommendations by the 
American Council on Education (ACE) or Dantes for military training and specialized licenses. During this process 
the student also compiles a portfolio of work examples and exhibits - artistic materials, reports, business plans, 
computer programs and the like. Students may take Thomas Edison State or Excelsior College examinations, 
utilizing credit awards in the process. This evaluation process often involves interviews about subject knowledge 
with a faculty expert to determine the type and level of credit that should be awarded for the student’s competency. 

SNL takes each student’s prior learning experiences and assigns the student to both an academic and a career 
professional advisor who then form a team with the student to put together a customized degree plan that can 
incorporate prior learning into a grid of fifty competencies. Twelve of those competencies comprise core courses, 
and the other thirty-six are divided into Arts, Humanities, Sciences, and career focus area competencies. 
Assessments of students’ portfolios, licensing examinations, military credits, externship and advanced projects, all 
become part of the degree process. The State University of New York’s Empire State College (SUNY ESC)’s 
competency-based MBA program integrates an online assessment of prior learning into portfolio development in its 
first residency course ( http://www.esc.edu/esconline/online2.nsf/htmEmba.html ). 

Taking the second or assessment approach to utilizing prior learned competency, WGU primarily requires students 
to demonstrate their knowledge, skills and abilities through taking the actual high-stakes assessments required of all 
graduates. Before doing so, they can take pre-assessments to determine their competency and deficiencies in various 
areas and receive mentor recommendations about how to best proceed during mutual development of the academic 
action plan. If students have sufficient college credit in various liberal arts areas, lower-division assessment 
requirements may be waived. Also, external certifications that are an integral part of the degree program that the 
student has already achieved, such as the SHRM certification for human resources majors or the CompTIA 
Security+ certification for network administration, are incorporated, thereby shortening the time for degree 
completion. Other programs similarly incorporate assessments — such as SNL’s required proficiency exams or 
demonstration of competency mastery through independent learning projects or portfolios. 

Advising and Academic Planning 

Since students vary in the amount of relevant competency that they bring to the degree program and differ in their 
educational goals, a large part of CBE involves developing with students an educational plan and continuously 
advising each student as he or she progresses toward degree completion. At DePaul University’s SNL, students 
receive academic advising throughout their degree program from an academic advisor and career advising from a 
professional career advisor; both remain with the student as a part of the student’s degree team throughout the 
student’s program, and the institution has developed several online tools to facilitate student planning and progress. 
These include a “dashboard” for administrative services, as well as an online grid to self-monitor their movement 
throughout their degree program. WGU assigns each student to a mentor who works with them throughout their 
degree program - from the creation of an individual academic action plan, through bi-weekly or more frequent 
communication, and advising about the competency development and preparation to demonstrate that through 
assessments. WGU has placed that AAP online in a similar dashboard format to facilitate enrolling in learning 
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resources, scheduling assessments, and tracking progress through the degree. SUNY ESC’s MBA program offers 
similar mentoring; an assigned faculty member works with each graduate student by providing assistance with credit 
for prior learning, the planning of the individual’s degree plan, and assistance throughout their course-taking 
experience at the college ( http://www.esc.edu/esconline/online2.nsf/html/mba.html ). 

Learning 

“Learning” is the colloquial term for the development of competency, the core involvement of students during their 
time at the University, since rarely do they embark on a degree being fully proficient in the field or discipline. 
DePaul University’s SNL offers credit-bearing courses in classroom, blended, and online formats for this 
competency development. SNL utilizes web-based and streamed technologies to facilitate student learning, as well 
as accepting transfer courses and enabling various learning projects ( http://snl.depaul.edu ). Once the WGU student 
and his or her mentor have determined their level of competency, the mentor counsels the student to take the set of 
learning resources that best fit their needs. An online “course of study” document (similar to a syllabus) lays out the 
recommended resources, sequencing, and timelines for completion so that students can maintain satisfactory 
academic progress. These learning resources can be third-party instructor-led courses arranged with other colleges, 
universities, or educational enterprises, but more often consist of independent learning resources (ILR) in which the 
student can begin at any time and progress at their own rate. ILRs include e-learning tutorials, web-streamed video 
instruction, simulations, and materials shipped to WGU student homes such as DVDs, CD ROMs, and workbooks. 
Use of the online library and textbooks continue to be the mainstay of competency development (Eastmond, 2006). 

Assessment 

Assessments within distance competency-based educational practice differ primarily if they are stand-alone (as is 
the case at WGU) or embedded within courses (as at DePaul University’s SNL). At SNL assessments vary based on 
the competency expert teams for that subject who determines the appropriate approach. There are four key 
assessment areas: the core degree courses area (termed Lifelong competencies); the Arts; the Humanities; and 
Career Focus Areas. These usually include coursework, final summative examinations, or project-based 
assessments in accord with DePaul’s SNL student-centered mission. WGU strives to include multiple measures to 
ascertain competency at the domain level; (several domains comprise a degree). These include objective and essay 
assessments delivered online and proctored at a testing center near the student’s residence. They also include 
performance tasks that demonstrate problem-solving or development of a professional product, such as a business 
statement or lesson plan. Assessments may also involve observations of student performance, such as in student 
teaching in a live classroom. Degrees at WGU culminate in capstone projects that integrate the learning from several 
domains into a substantive product and its oral defense. As mentioned earlier, Excelsior College and Thomas Edison 
State College deliver comprehensive examinations at a distance that result in college credit that can be used within 
their own institutions or utilized at hundreds of other colleges and universities. 

Challenges and Opportunities 

Competency-based education at a distance (or on campus) has yet to become the norm in higher education, yet the 
model is increasingly being recognized as an important educational reform and alternative means of adult learning. 
Certainly the regional, national, and association accreditation of CBE institutions, schools, and programs lend 
credibility to this approach. These CBE programs face similar measures of success as their higher education 
counterparts - that of enabling successful student experiences through recruitment, retention, academic progress, 
satisfaction, and graduation. Indeed it is the qualifications of graduates to perform the professional roles for which 
they have been prepared and their meaningful involvement as world citizens that form the measure of merit for CBE 
programs. SNL’s mission of student-centered learning and for its graduates to be global participants is demonstrated 
by the opportunities it offers its students in short international programs onsite in Asia, Africa, and Europe. These 
opportunities are often the first and only time that working and underserved adult students get a chance to travel 
abroad. WGU also measures its success in terms of its ability to serve groups such as low income, first-generation 
college, ethnic minorities, and rural students as fulfillment of its mission (Eastmond, 2007). 

One challenge confronting CBE is its acceptance in transfer between institutions of higher education and with 
employers for reimbursement. WGU has fastidiously worked to make the translation of its competency units into 
credit hour equivalents for this purpose, translating a pass as “the student has demonstrated competency at a grade 
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equivalent of B or better” (Eastmond, 2007, p. 10). Various accreditations have helped with this acceptance, as has 
various agreements with the Department of Defense, and major business partners that comprise WGU’s National 
Advisory Board. As the seventh largest private university in the United States, SNL has similar agreements with 
government, business, and international partners. Transfer of credit and degrees from one institution to another has 
been the prime reason for the European Union Higher Education Area’s interest in the competency-based approach 
(Colledo, 2007). 

Another challenge is harnessing technology in an effective way to deliver CBE through a distance anywhere in the 
world. WGU grapples with assessment delivery in secure, proctored locations overseas and has heretofore restricted 
its programs to active duty US military personnel who have access to such testing environments at bases and 
military installations. Similarly, the digital learning materials WGU utilizes for students to develop competencies — 
from electronic databases, e-books, various companies, websites, and open courseware — can seem fragmented until 
their interface is circumscribed into a common learning management system platform. As time goes on, new 
methods of digital security will provide avenues of further promise for student testing and assessment in online 
programs. Technology can continually be leveraged to facilitate academic discourse in learning communities and the 
interaction between mentors and their student advisees. 

Another challenge that is not unique to CBE is the accurate defining of learning units; however, the CBE approach 
holds the most promise of developing precise means of determining credit equivalencies. As Watkins and Schlosser, 
1992, explain the Carnegie unit (the basis of the credit hour) relies upon a teacher-centric model (i.e., 12.5 hours in a 
classroom with a qualified instructor equals 1 credit hour). The credit approach has evolved to include some 
dimensions of student work (e.g., 2-3 hours of outside work expected for each classroom hour), but is quickly 
becoming difficult to calculate given the plethora of instructional activities in which students are engaged in online 
learning that are independent of time or place. Based somewhat on these authors’ proposal of a student outcomes- 
based approach to defining credit hours, universities such as DePaul’s SNL and Western Governors University are 
continually re-defining competency-units (credit equivalencies) in a meaningful way. This approach, when refined 
and disseminated, may be emulated by other post-secondary institutions as they increasingly grapple with the 
meaning of credits. 

One of the basic requirements for education in the 21st century will be to prepare students for participation 
in a knowledge-based economy. . .(adult) students need new and different information resources, skills, 
roles, and relationships. The traditional educational model. based primarily on the concept of the school and 
the teacher in a classroom, standing aloneand not interconnected with society or any other educational 
institutions, will not generate competence in a knowledge society (Harasim, Hiltz, Teles & Turoff, 1996; as 
cited in Paloff & Pratt, 1999, p. 166). 

The promise remains for competency-based distance higher education to deliver affordable, accessible, quality 
education to working adult students regardless of location and at their convenience. The appeal of this model to 
accelerate time-to-completion for adults with competency and to ensure a level of proficiency for graduates seems to 
be growing throughout the world. As CBE programs and institutions graduate increasing numbers of capable 
professionals, the promise is being realized. 
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Redesign for Distance Learning Conversion: 

Case Study of a Graduate Course’s Evolution 

Linda J. Emerick, Ph.D. 

Introduction 

There are many challenges for course conversion from face-to-face to a distance learning format. This is 
true for conversions to fully online and conversions to “hybrid" courses — courses offered partially through distance 
learning and partially face-to-face. Both can present equally significant challenges. For example, university 
instructors have observed that online portions of courses require more time to administer and teach than face-to-face 
class sessions. They have also learned that approaches to online teaching, assessment, and interactions with students 
are influenced by communication that is primarily through email and downloads (DiBiase, 2000; Lazarus, 2003; 
Thoms, 2005). Educators must use a wide range of strategies to address the unique needs of online instruction. 

Some report using constructivist approaches that emphasize students’ interaction and collaboration. Others utilize a 
more structured approach with carefully worded syllabi and highly structured learning activities (Moore, 2002). In 
fact, the most recent report on online education in the United States indicates most universities believe “learning 
outcomes for online instruction are the same or superior to those for face-to-face instruction” (Allen & Seaman, 
2006). In addition to these general findings, there is a growing body of qualitative studies that have examined 
course conversion in more detail and depth. The qualitative research has investigated the different roles, processes, 
and dispositions that instructors have when designing and teaching online courses. It appears that the roles, 
attitudes, and considerations for design are quite different from those for face-to face instruction and are very 
complex (e.g. Barajas & Owen, 2000; Dewar, 2003; Donaldson & Conrad, 2002; Thoms, 2005). 

Conversion of courses for distance learning can be effective and, at the same time, complex. What must 
the designer/instructor consider when converting face-to-face courses to fully or partially online instruction? From 
the perspective of the instructor and the student, which forms of instruction best meet their needs? This study 
examined data from the redesign of a single face-to-face graduate education course through two iterations of 
distance learning. It followed the evolution of the course through conversion to a frilly online version and then to a 
hybrid version. The purpose of the study was to describe the processes, perceptions, and outcomes associated with 
the two conversions and provide insights for others who may take on this design challenge. The following research 
questions guided the study: 

1. What changes occur in objectives, learner outcomes, communication, and the roles and responsibilities 
of the instructor and students when face-to-face courses are converted to online and hybrid versions of a course? 

2. What are the perceptions and beliefs of students and the instructor regarding the strengths, weaknesses, 
and effectiveness of different versions of the course? 

3. What factors and considerations are critical when making design decisions about course conversions for 
distance learning? 



One Course, Three Designs 

The graduate course that was the focus of the study was named “Advanced Processes of Teaching and 
Learning.” Graduate students described it as the “most difficult course" in the Master of Education degree program 
for elementary and secondary teachers. While the content of the course was familiar to teachers (instructional 
models, teaching and learning theory), the processes covered were new to even the most experienced educators 
(front-end analysis, instructional systems design). Using their own classroom students and setting as a starting 
point, the teachers were introduced to the rigorous and highly analytical processes of instructional design and 
evaluation. Courses content covered the steps of identifying instructional problems in their school settings, 
conducting instructional goal analyses, learner and setting analyses, reviews of research, and designing effective 
learning solutions. Using data from their front-end analysis, all students in the class evaluated instructional models, 
adapted and modified the models, and developed new instructional solutions that addressed the original problem. 
Regardless of format, these were the basic competencies covered in each course version. 

The first iteration of the course was the on-campus face-to-face version. It consisted of 15 class sessions 
(45 contact hours over 15 weeks). The format included lectures, small group activities, group and small group 
discussions, and in-class work on projects and assignments. The instructor monitored student progress formally and 
informally through assessment of assignments and discussions in class sessions. The students had two texts for the 
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course, supplemented by handouts and additional notes from the instructor. Communication was primarily through 
face-to-face discussion and written feedback on assignments. 

The second iteration of the course was fully online and arose out of a need of the university. The instructor 
had not previously indicated a desire to convert the course for distance learning. In an expansion of international 
studies, secondary teachers in Shanghai, China participated in a version of the university’s masters degree program. 
The graduate courses were offered by university faculty either onsite in Shanghai or through conversion to fully 
online courses. The course, “Advanced Processes of Teaching and Learning”, was selected for online conversion 
(Hartzler-Miller, Emerick, & Kenton, 2006). Blackboard 6.0 was selected as the format for online instruction. The 
Blackboard site contained four learning modules that included written lectures, learning materials and activities, a 
variety of website links, and a discussion board forum. Students had one textbook to supplement the online 
materials. The course was 10 weeks in duration and all communications between the instructor and students were 
through e-mail and online documents. There was a “teaching assistant” located in Shanghai who helped the 
instructor monitor communications. 

The third iteration of the course was a hybrid format that included both face-to-face class sessions and 
Blackboard 6.0 online learning. Graduate students received 21 hours of face-to-face instruction (6 class sessions) 
and 25 hours of online instruction (45%/55%) over a 15-week period. The hybrid course included five online 
learning modules similar to those in the fully online course, online learning activities and supplemental materials, 
and links to a variety of websites. Students had one required textbook to supplement online materials. Face-to-face 
class sessions included lectures, discussions, and project work sessions. Communication between the instructor and 
students was through e-mail, document attachments to e-mail messages, and discussions in class. 

Methodology 

The study used both quantitative and qualitative methodology to describe and analyze the evolution of the 
graduate course, “Advanced Processes of Teaching and Learning”, through the different conversions. Data sources 
included: 

1 . Reflective notes of the instructor and interview data from a previous study 

2. Documents and notes from planning and design of the three versions of the course 

3. Student self-reports and open-ended questionnaires 

4. Student and instructor course evaluations 

5. Summary records of e-mail communications 

6. Course syllabi, lesson plans, learning modules, instructional documents and teaching materials 

7. Samples of student assignments and projects 

8. Summary course assessments, scores 

Content and inductive analyses were used to examine data from written documents, self-reports and 
questionnaires, interviews, and reflective notes. This form of analysis identified common characteristics and 
common themes within each version of the course. Descriptive statistics were used to analyze summary records, 
course and instructor evaluation reports, and other numerical data. Data were collected from three different 
semesters in which the different versions of the courses were offered. Each selected semester and course appeared 
to be representative of the three versions. The three course versions in the study were: 

Face-to-Face Version (Fall, 2004) Comparison of course evaluations, student responses to open-ended 
questionnaires, and sample assessments indicated that the selected class was representative of other face-to-face 
versions of the same course offered by the researcher between 2001 and 2005. Eighteen graduate students who were 
experienced elementary and secondary teachers enrolled in the course. All students successfully completed the 
course with overall grades of A and B. 

Online Version (Spring, 2006) Comparison of course evaluations, e-mail communications, and sample 
assessments from a study conducted of an earlier version (Flaitzler-Miller, Emerick, & Kenton, 2006) indicated this 
course was typical of the online conversion. Twenty-five graduate students living in Shanghai, China enrolled in the 
course. All were proficient in English and were experienced secondary school teachers and administrators. 
Twenty-three students completed the course successfully with overall grades of A and B. Two students withdrew 
due to health issues. 

Flybrid Version (Fall, 2006) Comparison of course evaluations, student responses to open-ended 
questionnaires, sample assessments, online communications indicated that the selected class was 
representative of previous offerings of the hybrid course. Seventeen graduate students enrolled in the course and all 
were experienced elementary and secondary teachers. All students successfully completed the course with overall 
grades of A. 
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Findings 



The initial examination of student ratings from course evaluations revealed little difference among the three 
versions of the course. When asked to rate the course for overall satisfaction and effectiveness on a scale of 1 to 5 
(1 indicating unsatisfactory, 5 indicating very satisfactory), students gave the face-to-face course a score of 4. 81. 

The online course received a score of 4.55 and the hybrid course was given a rating of 4.85. However, further 
analysis revealed several differences among the versions of “Advanced Processes of Teaching and Learning”. 

Evolution of Course Objectives 

As revealed in the course syllabi, the learning objectives for “Advanced Processes of Teaching and 
Learning” changed in each version of the course. In the face-to-face version, five learning objectives guided 
instruction. For the online and hybrid versions, there were four and three objectives respectively. Content analysis 
of the course objectives showed that those for the face-to-face course tended to be vague in wording 
(“Systematically analyze and evaluate educational situations using appropriate data collection. . .) and were not 
always aligned with a specific learning assignment or assessment. The objectives stated in the syllabi of the online 
and hybrid versions were more precise (“Systematically collect, analyze, and evaluate data about learning and 
content goals, learner, context and setting. . .”) and were aligned with the course assignments and assessments. In 
fact, wording from assignment directions and scoring rubrics were reflected directly in the objectives for both course 
versions. 

The conversion from face-to-face instruction to distance learning in the fully online course resulted in the 
elimination of the objective, “Engaging in meaningful reflection about ways to improve teaching and instruction. . .” 
Analysis of the instructor’s interview responses and notes indicated that the modification was considered necessary 
due to fears about limitations of technology for online learning, her own inexperience in managing online 
discussions, and concerns about language issues among the Chinese teachers. The objective reappeared in the 
hybrid version of the course. 

Students’ evaluations and comments about the course objectives supported the content analysis. Some 
students in the face-to-face course stated, “Your instructions for the assignments are clear but do not always match 
the syllabus” or “ The syllabus (objectives) could have been clearer”. They also gave the learning objectives the 
lowest overall rating recorded on the course evaluation: 4.23 (on a scale of 1 to 5 with 1 indicating unsatisfactory 
and 5 indicating very satisfactory). They scored all other items on the evaluation between 4.69 and 5.0. Students in 
the online course gave a rating of 4.42 for clarity and organization of the course but did not evaluate the clarity of 
course objectives specifically. Students in the hybrid course gave an overall score of 4.50 for the learning objectives 
and had no negative comments among their responses on the questionnaire. Their positive responses included 
statements such as, “What I was to do and why I was to do it were clear. Thank you.” 

The instructor’s perceptions of changes made to learning objectives in the three versions were mixed. 
Initially, she disliked changing the face-to-face course’s objectives to those for the online course. According to her, 
the modification had”. . .taken the soul out of the course. This was ‘slash and burn’”. As one of her colleagues 
expressed it, she believed the conversion became “all about the students being able to navigate the Blackboard site” 
and less about the original objectives of the course. However, the instructor’s beliefs changed when considering the 
objectives again for the hybrid course. The analysis and breakdown of the objectives into their basic parts from the 
online version provided the foundation for restating and realigning the objectives more carefully in the hybrid 
version. 

Evolution of Course Assignments and Learner Outcomes 

There were many differences among the three versions of the course for learner assignments and overall 
learning outcomes. Regardless of format, students were expected to understand the history and role of instructional 
design, identify instructional problems in their school settings or in case studies, conduct instructional goal analyses, 
gather data for learner and setting analyses, review and evaluate related research, and design effective learning 
solutions. To accomplish the latter, they had to demonstrate that they could evaluate and adapt or modify 
instructional models and develop new instructional solutions that addressed the original instructional problem. 

Findings from the content analysis of student assignments from the face-to-face version indicated that 
assignments were written with much detail and were personalized to the learner’s professional setting. The students 
chose issues from their professional setting and projects took on the aspect of “storytelling” with many examples 
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from the classroom. Over half of the front-end analysis assignments for the course were 15 or more pages in length 
as students wrote about their students and their schools. The guidelines for the assignment suggested 8 to 10 pages 
as sufficient. An example of this type of assignment content is shown below: 



I have taught fourth grade in the southeast area of XXX for ten years and am quite comfortable addressing 
the needs of a diverse student population. Managing a wide spectrum of academic abilities with a variety of 
instructional strategies has become a natural routine. This year, however, I have a new position as facilitator in a 
different elementary school. My principal is also newly appointed to her administrative job. Our school has a 
unique history. In the past, it has been an alternative school and a life-skills adult functional school. Families in the 
neighborhood began a battle to get their elementary school back. The only route that was found to be successful was 
to make the school a magnet school. At present, XXX School not only has a 3 and 4-year-old Outreach Special 
Education Program, a self-contained kindergarten class, and a primary adaptive class, but also has the magnet focus 
of multiage classes. 

Neither my principal nor myself have had any experience teaching in a multiage school, but we do have 
somewhat of an understanding of what it should look like. Flowever, in our first weeks of acclimating ourselves to 
the students, staff, scheduling, and families, we realized that there was an overall misunderstanding or complete lack 
of knowledge of the intention of the multiage magnet program. Parents who have had to apply for their child to 
participate in the magnet program are unaware of the multiage homeroom setting, questioning why kindergarteners 
are traveling to lunch with first-graders. 



The content analysis of corresponding assignments from the online version indicated a major change from 
those of the face-to-face version. This was largely due to changes in instruction. Instead of choosing their own 
instructional problems for the assignments, the online students used a single case study presented by the instructor. 
For example, instead of completing the front-end analysis assignment for their school or classroom, they applied the 
analysis skills to the case study of the fictitious “Mrs. Albert” and her high school English classes. Instead of using 
“storytelling” and examples from real life, the students’ assignments were presented as individual questions that 
mirrored the process to be learned. The example below illustrates how the review of research assignment was 
changed for the online version. 



Mrs. Albert is preparing to conduct a search of research articles and expert opinion related to her 
instructional problem and front-end analysis. Review the information she has gathered so far in Steps 1, 2, and 3 of 
the front-end analysis (goal analysis, learner analysis, setting and context analysis). Answer the items below: 

1. What are three to four categories of research Mrs. Albert might investigate to learn more about her goal, her 
learners, or her setting and context? Explain why each category is appropriate to investigate. 

2. Mrs. Albert has found 3 articles that might relate to her instructional problem. The research/expert opinion articles 
are located in this section of Learning Module 2 ( Cool Influence , Core Curriculum , and Gender Differences ). Skim 
each of the 3 articles. Which one article would you personally select as most relevant to Mrs. Albert's needs 
assessment? Give the title of your choice below. 



3. Why did you choose the article you selected in #3 above? Write a brief explanation of why you think it is the best 
for Mrs. Albert. 



Student assignments and products from the hybrid version were similar technical report — brief, accurate, 
but with explanations and examples. For example, content analysis showed that the majority of students’ 
instructional solution assignments did not include “storytelling” and wandering explanations. Instead, they wrote 
responses to the different parts of the assignment, providing brief descriptions and examples in an objective manner. 
In comparison, the assignments were concise but provided much more detail than assignments from the online 
version. An example follows: 
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The instruction for the solution will take place in the form of a unit of Language Instruction during the class’s 
Language Arts Block with Miss XXX. This is appropriate because the students are with Miss XXX at this time and 
writing and grammar are part of the XXX County Public Schools curriculum. The unit of instruction will span five 
to seven days. There will be five 45 -minute lessons taught during the unit. Each lesson will be presented from 
10:00a. m until 10:45a.m. Many students receive reading interventions at various times during the morning. During 
this time Miss XXX has all the students present in the classroom before they transition to another teacher for math 
and science instruction. The instruction will take place in the students’ classroom which is a trailer located behind 
the elementary school. It is important that learner’s feel comfortable in their surroundings and that they have all 
materials accessible to them during instruction (Lewis, 1998). The students have been learning in the trailer for the 
duration of the school year. This indicates it is the best environment for the intended instruction. The lessons 
planned for this unit will be tied into the students’ first writing composition. At the beginning of the school year the 
students wrote a personal narrative about a time they lost something in accordance to VSC indicator 4.A.2.a, which 
requires that students compose written presentations to express personal ideas. 



Again, analysis of assignments, syllabi, and assignment guidelines indicate the changes resulted from 
differences in instruction and how course requirements were explained. While students in the face-to-face version 
received extensive verbal directions from the instructor and limitless opportunities in class to ask questions and 
develop their assignments, students in both the online and hybrid versions relied more on online directions and 
examples. For the students in the hybrid version, instruction for course content and guidance for assignments were 
delivered in four ways and the assignments were integrated throughout. First, the online learning modules provided 
content, activities, and guidelines about the assignments, blending both real life application and a case study 
example. Second, in-class lecture and discussions reinforced the learning modules and addressed any student 
confusion. Third, structured rubrics provided a step-by-step framework for the student to begin his or her 
assignments. Finally, samples of the assignments completed by former students in the course, along with 
annotations by the instructor, were available as examples. 

Students’ perceptions of assignments and what they demonstrated varied among the three versions. 100% 
of the students in the face-to-face version gave a rating of 5.0 (on a scale of 1 to 5 with 1 indicating unsatisfactory 
and 5 indicating very satisfactory) for “assignments relating directly to their profession”. Similarly, they gave an 
overall rating of 4.92 for “assignments being appropriate for demonstrating their new knowledge”. Additional 
comments from questionnaires and evaluations indicated that the students’ also valued “having a choice of focus for 
my assignments.” However, several also complained that “assignments are not clear or seem to change” and “there 
is too much to learn in too little time.” 

The perceptions of students in the online version were more difficult to discern. 88% gave a rating of 4.4 
for “assignments are evaluated fairly” and 91% rated “accomplishment of learning goals” at 4.5. There were few 
comments through e-mails or questionnaires about the usefulness of the assignments although a few hoped to use the 
information in their schools in the future. A few also indicated they were not sure they would use the information 
again. 

Students in the hybrid version gave ratings for assignments slightly below those of students in the face-to- 
face version. 100% of the students in the face-to-face version gave a rating of 4.8 (on a scale of 1 to 5 with 1 
indicating unsatisfactory and 5 indicating very satisfactory) for “assignments relating directly to their profession”. 
100% also gave a rating of 4.8 for “assignments being appropriate for demonstrating their new knowledge”. They, 
too, responded similarly to those from the face-to-face version and identified “ having a choice of what I can do for 
my projects” and “it was a real application of what we are learning” as best features of the their course assignments. 
Unlike the other course version, they made no negative comments about assignments. 

Finally, the instructor had her own beliefs about the quality of the course assignments and demonstration of 
student learning. Initially, she believed the assignments for the online version were “only a faint version of what the 
other students experienced. There was little emotional connection between the students and their assignments.” 

This was partially supported by responses from students on course evaluations. The Shanghai teachers gave their 
lowest overall rating of 4.17 for “enthusiasm for the course.” This was low in comparison to the other two versions 
of the course. However, after analysis and revision to incorporate online learning, she stated that the assignments 
for the face-to-face version did not demonstrate the learning she had hoped for. In the end, she believed the 
structured format of the hybrid version produced the most useful and effective learning experiences and assignments 
for the graduate students. 
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Evolution of Communication 



As expected, each of the three versions of the course had very different forms and quality of 
communication between students and the instructor. However, the reactions of students to the different forms of 
communications were sometimes surprising. 

For the face-to-face version of the course, all communication between the students and the instructor 
occurred during class time through discussion and one-on-one consultation. Students could also communicate with 
the instructor by e-mail but most waited for the weekly class session to ask questions and discuss assignment issues. 
Communication also occurred when the instructor wrote comments on assignments returned to students as part of 
the evaluation process. Samples of assignments indicate that often the comments were lengthy and detailed. 
Analysis of course evaluations and student questionnaire responses indicated that quality of communication was an 
important feature of the course. Students gave a cumulative rating of 4.77 for “responsive communication from 
instructor” and most frequently made comments were about “good instructor interaction and support” (mentioned by 
53%) and “ good interaction and communication with other students” (mentioned by 35%). However, there was a 
drawback to the flow of communication. 25% of students who submitted comments about negative aspects of the 
course mentioned “instructor spends too much time on the ‘whiners’ and ‘goofy people’” and that class time was 
“wasted by discussions that go on too long.” 

There were three modes of communication for the online version. The primary mode was e-mail 
messaging. The second mode, from instructor to students only, was through announcements and information on the 
Blackboard 6.0 site. The third mode was also through e-mail but included written edits and comments on 
assignments that the instructor returned to the students via e-mail. An analysis of the e-mail communications from 
students to the instructor showed that most inquiries were to confirm receipt of assignment attachments, ask for 
extensions to complete assignments, and report technical problems with accessing online learning modules. There 
were few questions about course content or assignment directions. However, the students rated quality of 
communication for the course both high and low. 88% of the students rated quality of communications between 
them and the instructor at 4. 1 and 18% gave a rating of 3.0. In contrast, 92% stated that the rate and quality of 
responses they received from the instructor were “excellent and helpful” (4.5). At the same time, some e-mail 
communications indicated the students “would like to see the instructor” and asked when the instructor would visit 
them in China. In comments about the course, students selected “communication” as the area in greatest need of 
improvement for the course. 

For students in the hybrid version of the course, communication was a blend of the written and verbal 
communication and in-class discussions of the face-to-face version and of the Blackboard 6.0 site and e-mail 
exchanges of the online version. Students in the hybrid version frequently e-mailed the instructor with questions 
about course content and assignments. It was not unusual for 10% to 20% of the students to submit drafts of 
assignments and ask the instructor to “Please read what I have so far and let me know if I am on the right track”, 
depending on the assignment. There were also occasional e-mail requests for extensions and technical help. With 
fewer class hours, it was expected that students would rate direct communication with the instructor and classmates 
at a rate lower than students in the face-to-face version. However, 100% rated instructor/student communication as 
“very good” to “excellent” (4.87). 93% of the students believed that there were excellent opportunities for “open, 
respectful discussion” (4.73) in the class. Content analysis of responses to open-ended questions revealed no 
comments about quality of communications. 

The perceptions of the instructor were aligned with those of the students in the three versions. However, 
she believed that her communication with students about their completed assignments took on a new and more 
important role in both the online and hybrid course versions. Analysis of sample assignments indicated that she 
wrote more comments and used evaluation of the assignments as a teaching tool to a greater degree in the online and 
hybrid versions. In the face-to-face course, comments and responses to students were general — “good work on this 
part”, “this might need more detail.” Comments to students in the other two course versions often explained 
problem areas and gave examples for improvement. The instructor believed she spent more time communicating 
with students through e-mails and through comments on student assignments for the online and hybrids versions. 
However, she did not document time spent in communication. 

The Evolution of Student and Instructor Roles 

The greatest variations among the three versions of the course occurred in roles of the graduate students 
and the instructor. The face-to-face version provided the baseline information for the comparison among the three 
versions. 
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For the students in the face-to-face course, the instructor was the primary source of information and 
instructional support. This was evidenced by the total amount of face-to-face instruction that happened in the course 
(45 hours) as well as the amount of lecture, and the amount of group discussion that took place. Students did not 
view this as negatively and instead rated the course favorably for “instructor feedback and interaction with students” 
(4.92) and “offering a stimulating and challenging course” (4.92). It was important to the class members that the 
instructor was available to address every question they had and clarify assignments and content. Content and 
inductive analysis of student comments on questionnaires found the themes of “instructor is helpful to me” and “the 
instructor is very knowledgeable” as the two most frequently mentioned positives in this category. One student 
explained the perceived instructor role in this way: “ I thoroughly enjoyed the class and the only problems I had 
were my own that kept me from concentrating 100%. But [the instructor] was always there to encourage me, to 
keep me going on.” 

There was a dramatic shift in the roles of the students and instructor in both the online and hybrid versions 
of the course. In both instances, online learning modules written by the instructor became the vehicle through which 
students managed their own education. The instructor, by both her own and the students’ assessments, moved into 
the role of a facilitator for these two versions. By design, the students were given broad timelines for completion of 
learning modules and assignments. There was little communication from the instructor during these periods except 
to direct inquiries from students. Students were expected to decide when and how to conduct their learning and 
assignments while the instructor was available to provide feedback. Students in the online version of the course 
expressed concern about managing their own learning in responses to a pre-course survey. They had never 
participated in an online course before and were uncertain what to expect. However, the majority adjusted well to 
their new role by the end of the first learning module as evidenced by the decrease in the number of late 
assignments. There were seven late assignments for the first learning module and only three for the second learning 
module. As the course progressed, only one student continued having difficulties managing his own learning. 

Students in the hybrid version of the course ranked the “online learning modules” and the “ability to work 
independently” as the two greatest strengths of the course. Their responses and comments cited “ability to self- 
manage” and “my choices for what and how I am learning” as primary reasons why the hybrid version was effective 
for them. They gave an overall rating of 4.93 for the quality and effectiveness of the course. In addition, while the 
majority liked the online portions of the course, 100% of the students stated during a group interview session that 
the six face-to-face class sessions were an important as part of the class. 23% of those responding thought perhaps 
only 4 to 5 class sessions were necessary. They stated that they liked communicating other students and having 
questions answered by the instructor in person. 

The instructor saw major changes in her role among the three versions. She believed she changed from 
“dispenser of knowledge and information” to colleague and facilitor. Previously, in the face-to-face version, the 
instructor believed she dominated both class time and decisions about student projects. The conversion of the 
course to a distance-learning format forced her to shift the responsibility for learning to the students. In her opinion, 
the hybrid version provided the best opportunity for the students to function as independent, adult learners with the 
instructor shifting emphasis to the student as the decision-maker. At the same time, it seemed to satisfy her personal 
need for interaction with the students. “I did not realize how much the force of my personality was a factor in the 
course until I had to step back, away from the students in the online and hybrid courses.” 

Conclusion 

As researchers have noted, the roles, attitudes, and considerations for the design of distance learning 
instruction are quite different from those for face-to face instruction. This study revealed the complexities of design 
by following the evolution of a single course. The findings indicate that many, varied factors play a role in the 
successful conversion of a face-to-face course to an online or hybrid version. While students gave high marks for 
effectiveness to all three versions of the course, data revealed subtle differences. These differences were evident 
from the qualitative data. They included shifts in attitudes and perceptions of the learners and instructor toward each 
version, variations in learning objectives, changes in formats and quality of assignments, challenges in 
communication, and a move to more student-centered learning. It was clearly demonstrated that the beliefs of the 
instruct or/designer also play an important role in making a successful conversion. The instructor’s personal 
preferences and assumptions about learning appeared to initially block objective consideration of distance learning 
possibilities for the face-to-face version. Careful analysis of the course components, information from students, and 
examination of learning outcomes led to improved design for a hybrid version of the course. 

Findings from this study are not generalizable and apply only to the instructor, students, and events for 
conversion of this particular course. However, revelations about the three versions of the course may help others 
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who are attempting distance learning course redesign. Several key questions are critical for the instructor who, as 
the course designer, is making decisions about conversions for distance learning. 

1. Has the designer conducted an objective, in-depth analysis of the course to identify opportunities for distance 
learning? 

The instructor in this study would not have compiled data necessary for the redesign of the original course 
under ordinary circumstances. After all, the student evaluations indicated it was a sound course. However, an in- 
depth analysis of all features of a face-to-face course can help the designer identify areas of improvement and 
opportunities for distance learning in an objective manner. 

2. Has the designer contemplated changes that would be necessary for the course to incorporate distance learning? 
What would be gained? What would be lost? 

Learning objectives, assignments, and communications for the face-to-face version of the course underwent 
major revisions to incorporate online components. There were concerns that the intent and quality of the face-to-face 
course could be compromised in the conversion. The designer must weigh the value and impact of necessary 
changes before making conversions for distance learning. 

3. Has the designer anticipated his or her needs and the students’ needs for professional communication and 
personal interaction? 

Findings indicated that, regardless of format, graduate education students who are working professionals 
prefer interaction with others in their profession. The majority of students in the face-to-face version and the hybrid 
version valued and utilized discussions and dialogs (face-to-face and online) with the instructor and colleagues. 
Students in the online version preferred more personal communication with the instructor. In course conversions, the 
designer must consider how to maintain the quality of communication within a distance learning environment. 

4. Has the designer considered how course conversion might impact his or her method of instruction? 

Findings of the study demonstrated that the personality of the instructor had become the focus of the face- 
to-face version. The conversion of the course to online and hybrid versions placed the students at the center of their 
own education. By providing more opportunities for online instruction, the instructor became a facilitator and the 
students became responsible for managing their learning. Designers must consider the impact, positive or negative, 
that conversions to distance learning may have on the role of the instructor and the students. 



142 




References 



Allen, I.E. & Seaman, J. (2006). Making the grade: Online education in the United States, 2006. Needham, MA: 
Sloan Consortium. 

Barajas , M & Owen, M. (2000). Implementing virtual learning environments: Looking for holistic approach. 
Educational Technology & Society 3, 39-53. 

Dewar, T. (Ed.) (2003). Keyboard voices: An e-book about facilitating online. Available through 
www.calliopelearning.com 

DiBiase, D. 2000. Is distance teaching more work or less ? The American Journal of Distance Education, 1 4, 5-20. 

Donaldson, J.A. & Conrad, R. (2002). Moving F2F activities online. Paper presented at the 18 th Annual Conference 
on Distance Teaching and Learning. Available through www.uwex.edu. 

Hartzler-Miller, C.; Emerick, L.J.; & Kenton, J. (2006). Learning “Repackaged”: Transformations from Direct to 
Distance Instruction. In Society for Information Technology and Teacher Education (SITE) 17 th Annual 
Conference Proceedings. Chesapeake, VA. 

Lazarus, B. D. 2003. Teaching courses online: How much time does it take? Journal of Asynchronous Learning, 7, 
47-54. 



Moore, J.C. (2003). Elements of quality’: Synthesis of the august 2002 seminar. Needham, MA: Sloan Center for 
OnLine Education. 

Thoms, K.J. (April, 2005). Kicking and screaming: Overcoming faculty resistance to teaching online courses. Paper 
presented at the Tenth Annual Mid-South Instructional Technology Conference, Murfreesboro, Tennessee. 



143 




AT A DISTANCE: A Model for Distance Education 



Amber D. Evans 
Barbara B. Lockee 
Virginia Tech 

ABSTRACT: Trying to shoehorn “traditional” instruction into a distance education format ignores key 
features of distributed learning environments. Designers and educators of distance education have some 
knowledge of how their instruction will be delivered — what “must” be used. The AT A DISTANCE model, 
and acronym for seven identified key stages, explicitly incorporates contextual issues into the early stages 
of systemic instructional design to select the most appropriate technologies available to build effective 
distance education programming. 



Introduction 

Instructional design (ID) offers a systematic process for ensuring the development of effective learning 
environments. In the world of distance education, that same desire applies. The creation of learning solutions 
through ID is typically based on a model that serves as a framework for the design and development process. While 
distance education reflects a specific context for which instructional programming is produced, it maintains inherent 
features that require a customized model to guide development for this delivery approach. 

As found in the “traditional classroom,” designers and teachers want to be sure that their students are 
receiving quality education. As Head, Lockee, and Oliver (2002) found, unlike traditional classrooms distance 
education presents a myriad of different (and sometimes new or difficult) media or modes in regards to how the 
instructional program has to be delivered. 

Consider the technologies that may be in use. What if a particular system of providing distance education 
has limited (or non-existent) face-to-face interactions? Do time delays exist among members of the learning 
community? Is the targeted class synchronous, asynchronous, or a blend of both? Professors teaching in distance 
education environments are aware that there are other complexities as well: what technologies are available, how 
easy are they to use, what are the uses; what is possible, probable, unlikely, or impossible to do? These 
considerations should factor into how materials will be organized, developed, presented, delivered, and ultimately 
designed and tested for maximum learning effectiveness. 

Distance education does not offer a new or better way of teaching or learning, it is a different context that 
provides another approach to teaching and learning. As Gustafson and Branch (1997) stated, “[t]he greater the 
compatibility between an ID model and its contextual, theoretical, philosophical, and phenomenological origins, the 
greater the potential is for success in constructing effective learning environments” (p.16). If the model can be 
aligned to the way it is going to be used, the instructional designer will be more likely to create a successful learning 
experience (in any medium). The same is true for distance education. The model by which distance courses are 
developed must consider the features of this specialized learning environment. 

The model consists of seven key stages, four of which have sub-stages of their own. This model, called AT 
A DISTANCE, is an acronym for the primary seven stages: Analysis, Technologies, Affective domain. Design & 
develop, Implement, Sample, Tryout, Adjustments, Negative consequences, Completion, Evaluation & 
endorsement. The foundational concept for the AT A DISTANCE model was drawn from the following existing 
models: the ADDIE model (Rothwell & Kazanas, 1992), the ARCS model (Keller, 1987), Gustafson and Branch’s 
(2002) suggested models for an organizational setting, and Mager’s (1997) Performance Analysis flow diagram. 
Combined aspects from each of these established and respected models have resulted in the current model that 
provides a hybrid solution to the production and systematic approach to distance education. 

What makes this model different from many others is that it immediately acknowledges the significant 
influence that administrative or existing infrastructure places upon the designer or instructor in choosing how to best 
design for the delivery system in place. According to Gustafson and Branch (1997), “[mjodels also assist us in 
selecting or developing appropriate operational tools and techniques as we apply the models” (p. 21). Related to this 
principle, any designer or teacher would prefer to have full control over what, how, and why they would choose to 
use a particular development tool, media, or mode to deliver their instruction. The reality is that an instructor is 
presented with a list of available technologies and told to “pick one or several” to deliver their instruction. It may 
seem like a step backwards, and in many situations, it is. Often being “stuck” with a particular technology that does 
not match the instructional goal will result in an ineffective instructional experience. 
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An analogy may help: imagine spending an enormous amount of time developing an exquisite-looking 
square peg and then being told it will need to fit into a round hole. Parts of the peg will need to be stripped away to 
make it work, or one may have to begin again to create the properly fitting round peg. The goal of the AT A 
DISTANCE model is to help designers see the round hole at the beginning and to build peg(s) appropriately while 
(hopefully) reshaping the hole at the same time. Through early recognition of the technologies and tools the 
instructors are “bound” to use, instructional designers can create a more complete and cohesive learning solution. 

A Closer Look: “AT A DISTANCE” 
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As mentioned earlier, AT A DISTANCE is an acronym for the primary seven stages of this proposed ID 
model: Analysis, Technologies, Affective domain. Design & develop, Implement, Sample, Tryout, Adjustments, 
Negative consequences, Completion, Evaluation & endorsement. The AT A DISTANCE model is about taking 
established systematic design of instruction with clear performance objectives in mind and selecting the appropriate 
technologies or tools to build learning that appeals to the learner’s affective domain. From there the coursework is 
developed in stages of modules and units. At the stage of implementation, a prototype module or unit is created and 
tested. Adjustments are made at which point the designer steps back and asks, “Are there negative consequences in 
having my learners do what I’m asking them to do?” Depending upon the answer, the prototype is either further 
adjusted or tested. When satisfactory, the next sample (module, unit, etc.) repeats this process. This continues until 
all prototypes are satisfactory in which case the course can then be completed where a final evaluation of the course 
is conducted before moving on to endorsing the final product (which in this case would be the course in its entirety 
or a curriculum as a whole). 

Analysis 



Borrowing from the Analysis phase of Rothwell and Kazanas’ (1992) ADDIE model, the analysis stage as 
part of AT A DISTANCE is a look into the audience’s characteristics (needs/desires), the content to be taught, and 
the context in which it will be held. The level at which these three parts receive treatment will vary. Therefore, an 
in-depth review of the audience may or may not be required. As Gustafson and Branch (1997) had determined 
within a given classroom of “typical classroom instruction” for a college course: 

Most teachers assume (with real justification) that students will be assigned to or will enroll in their classes 
and that there will be a specified number of class meetings, each of a pre-determined length. The teacher’s 
role is to decide on appropriate content, plan instructional strategies, identify appropriate media, deliver the 
instruction, and evaluate the learners . . . teachers usually need to identify and adapt existing resources 
rather than engage in original development, (p. 38-39) 

In the case of a distance education course, however, a closer look into the demographics of the target group 
may be pertinent to development. Consider that the audience is likely to be broader in age, experience, and locale. 
What discrepancies exist? Will any of them require special services or alternate accessibility? If nothing else, be 
sure to understand what the learner-related characteristics are for the target audience. Find out what prior 
knowledge, skills, abilities, and attitudes they may have. If possible, also determine audience members’ ages, 
gender, races and what their general aptitudes may be. (Rothwell & Kazanas, 1992). 

After the audience has been defined, one must analyze the instructional content. This activity is defined by 
Gibbons (1977) as “the process of breaking large bodies of subject matter into smaller and instructionally useful 
units” (as quoted in Rothwell and Kazanas, 1992, p. 133). The resulting process of this allows designers to proceed 
with the development of instruction. 

Lastly, an analysis of the learning context needs to be performed. Designers need to be aware of the 
settings in which the instruction will be taking place. This simply means being aware of wherein the instruction will 
be engaged and the actual environment of the learned material when applied outside of the instruction. 

With these three sub-parts completed, the next step is to write the performance objectives. Here, designers 
determine exactly what it is that learners will be able to do as a result of the instruction (or module, or unit, etc.). At 
this point in the AT A DISTANCE model, note that the performance objectives are connected to the Affective 
domain segment of the model. Before learner’s attitudes are addressed, the next stage in the proposed model will be 
discussed. 

Technologies (Tools) 

Designers and instructors are expected to be proficient not only in knowing what technologies can do for 
them instructionally, but also how to proficiently utilize such technologies for distance course development and 
teaching. With only so much time and so much software available, it is nearly impossible to fully understand how 
or what is the “best” media is. There is media for development. There are media and tools for delivery. There are 
still different ones for presentation and others still for activity or engagement among learners. With so much to 
choose from and utilize, it can overwhelming. 

For this reason, this stage is an important one to incorporate into the process of developing a distance 
course. Expect to spend a fair amount of time either working with someone knowledgeable or doing a fair amount 
of research to determine which methods (and media) will work best for the intended learning outcomes. 
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This stage is often the most frustrating — yet interesting — segment of the process. As part of this process, 
designers may need to assess whether or not media selection choices will need to be adjusted. Gustafson and Branch 
(1997) illustrate, “| i Instructional development models can directly or indirectly specify products, such as time lines, 
samples of work, deliverables, and periodic endorsements by appropriate supervisory personnel. While models 
provide the conceptual reference, they also provide the framework for selecting or constructing the operational tools 
needed to apply the model” (p. 24). When done properly, the end result is learning that aligns with the original 
desire of the instructor. 

Affective (Domain) ARCS 

The affective domain is often regarded as a difficult aspect to address in the ID process. Engaging 
education and learning is thought to be important because it draws the learner into the situation, making them 
focused and active in the instructional program. 

This is why the third part of the model is important to the design of distance education. Keller’s ARCS 
model (1987) is also an acronym for: Attention, Relevance, Confidence, and Satisfaction. The inclusion of the 
ARCS model is designed to draw attention to the how and why’s of the actual design and development of the course. 
Tied into the performance objectives, it ensures that the desired performance is relevant and can be met with 
confidence in the end. If the ARCS model is connected with the Technologies segment of the AT A DISTANCE 
model, then designers can assess whether or not the presentation and delivery of the material maintains attention and 
gives the learner enough feedback or interaction to instill a sense of confidence in their learning. Once it has been 
established that the analysis is complete and that the technologies are suitable and the affective domain has been 
given address, designers can begin planning and developing the actual instruction. 

Design & Develop 

This stage is common in all ID models and needs little explanation. It is where all the previous stages come 
together in the building of cohesive units and modules, as part of a course or larger curriculum. This will vary 
widely as much of it will be dependent upon what technologies were selected and what outcomes were desired. 

Some materials may be newly developed (taking the longest) while existing materials may be re-purposed into a 
different format that is more suitable for the distance learner to use and understand. As these parts are close to 
completion, the next stage of the AT A DISTANCE model can begin. 

Implement 

In this stage, foil implementation of the distance course is not yet intended, as in the ADDIE model. 

Instead, this stage is a form of formative evaluation, allowing for a Sample to be created, tested, adjusted, and tested 
again. The idea is that in this stage, the implemented product will be pilot tested in a similar learning environment 
as the intended instructional setting. This process would allow for the most useful information to be fed back into 
the revision of the unit or module. Multiple versions and multiple modules or units can move through this segment 
at any time for as many times as needed to perfect the learning experience (en route to obtaining the desired 
performance objectives). 

Sample -> Tryout -A Adjustments. 

As sub-parts to the Implementation stage, these steps work together to identify a functional draft, prototype, 
or deliverable to then be tested in a realistic setting. The idea is to be as close to the “real deal” and to obtain 
feedback that can then be incorporated into the revised version of the instructional program. Once the sample item 
has passed the Tryout and Adjustments sub-stages, it can then be reviewed in the next primary stage. 

Negative Consequences? 

Are there negative consequences? This segment of the AT A DISTANCE model is borrowed from 
Mager’s (1992) Performance Analysis flow diagram for performance improvement. In this segment, Mager asks if 
what learners are asked to do is punishing to them. For example, if learners are required to engage in some wiki 
board discussion, but the discussion board silently “times out” while they are working on their revisions, then the 
users lose all of their work when they attempt to submit or reload the page. When this happens, this is a negative 
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consequence of the technology (or perhaps even in the learning itself) that is detrimental to the learner’s attitude and 
instructional experience. In short, learners were doing as instructed, and actively engaged to the point of being 
unaware of being “disconnected.” When attempting to “do good” and turn it in, they lost all of their work, 
discouraging them from wanting to do the same next time. Removing these problems early on also helps to 
establish the look of a complete and finished product. 

Completion (of Course/Curriculum) 

After the course has passed the Negative Consequences review and appears to be functional and engaging 
(or fun), then the unit or module can be developed in completion. Each unit or module can be completed at different 
times and plugged together in the end to form a cohesive course ready for learning. 

Evaluation & Endorsement 

The course is finished and all that remains is to conduct evaluations. Referring to Kirkpatrick and 
Kirkpatrick’s (2006) four levels of evaluation, you could determine at what levels the course meets the requirements 
and at what levels your students are also assessed at. In this case, the course evaluation is of primary concern and it 
should be noted that feedback from students and other designers would be ideal in determining the effectiveness of 
the overall course. When an evaluation has established that the course is complete and meets all objectives and 
finishes up with learners who can transfer their knowledge, the course is ready for teacher or departmental 
endorsement. 



Conclusion 

Why use AT A DISTANCE at all? Gustafson and Branch (1997) stated: 

Systems-oriented models, such as those created by Branson (1975), Dick, Carey and Carey (2001), and 
Smith and Ragan (1998), typically assume a substantial amount of instruction will be created, such as an 
entire course or entire curriculum. Substantial resources are typically provided to a team of skilled 
instructional developers and subject matter experts. Whether or not original production or selection of 
materials will occur varies, but in many corporate settings original development may be required. 
Assumptions about the technological sophistication of the development and delivery systems also vary, 
with the decision often being based on the infrastructure available for course delivery. The amount of 
front-end analysis is usually high, as is the amount of tryout and revision. Dissemination and utilization 
may be quite wide, but probably does not involve the team that did the development, (p. 36) 

As part of addressing the wide-use and far-reaching capabilities of distance education, AT A DISTANCE 
recognizes that many people are involved in the whole process of distance education. The instructor or designer 
creates the materials, but there is also policy, infrastructure, and technologies that also need to be taken into account 
and appropriately included within the whole scope of developing effective education for lasting learning. 
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The roles of design: a new method of instructional design 

Brad Hokanson, Charles Miller, Simon Hooper 
University of Minnesota, Penn State University 

Introduction 

This writing addresses the perception and activities of the instructional designer. At its core, this is an examination 
of beliefs, of values in the field; the questions we ask concern aesthetics, innovation, and the very nature of design. 

It is a theoretical investigation of our perception of ourselves as designers. "Men often define themselves through the 
skills they acquire, and the issues to which they put them" (Dreyfus & Dreyfus, 1986, 11). Most in the field of 
instructional design today would describe themselves as seeking to be either an engineer or scientist. (Visscher- 
Voerman & Gustafson, 2004). 

We begin with addressing the current state of instructional design, which, by many accounts, is limited by its 
approach to design issues. Most in the field function as and seek to be instructional engineers. The driving force in 
instructional development often is the efficient production of instructional materials, which limits innovation. The 
current practice in instructional design concentrates on completion rather than quality. There is an inherent 
difference between design and production. Design is what happens past done; more than being a simple act of 
production, design is one of creation and innovation, of seeking of quality in all aspects of the process. 

We will introduce a new conceptualization of the process of instructional design, replacing the worn steps of 
ADDIE with a series of exemplars for design practice; the instructional artist, instructional architect, instructional 
engineer, and instructional craftsperson. As a sequence of leaders, or as a representative design team, this "role- 
based" design method will encourage innovation and higher quality design. 

Why has the field of instructional design failed to move forward? That is, forward with new ideas, forward with 
innovative methods of teaching using technology, and forward with advanced theories of the use of educational 
technology. We argue that the focus of the field since inception has been on lower level learning, often building 
work that centers on demonstrable content memorization as opposed to the more complex and advanced learning of 
thinking skills that is needed for the world today. As instructional designers, and in terms of learning, we aren’t 
rich, we’re simple. We aren’t authentic, we’re removed from context. We don’t advance; we recreate the teaching 
and instructional methods of the past century. 

Central to this failure of instructional design is our methodology of design. Most instructional designers, when 
questioned about their use of a design method, have a one word answer: ’’ADDIE”. Clark (1995) held that learning 
from instruction media would only change when the method changed, regardless of media. Our limited success in 
designing instructional media comes when we are all using the same method of design. Logically, our design will 
only change when our design method changes as well, regardless of the medium of our work. 

Examining the revered method of ADDIE does not provide an argument for continued use. It is not an invented 
method for planning and designing in instructional design, but rather a description of established and vernacular 
practice. It appears to be first described in a number of sources as a description for what actually happens in many 
design fields, only later to be formalized through the literature of instructional design (Molenda, 2003). 

In reality and in common use, the ADDIE models differs very little from codified design models used in other fields, 
notably architecture. Architects commonly use the terms schematic design, design development, construction 
documents, and contract administration to contractually segment the process of design and constructing a building. 
However, while the process can be divided into these phases (along with subsequent client billings), within good 
architectural firms, almost no designer limits or constrains their design activity within these steps. For example, 
design still occurs during construction, often adding to the value of the design. Architecture has a rich history of 
design methodologies looking beyond simple steps, both in terms of codification, and in terms of analysis and 
evaluation before and after the formal design process (often called “pre-design” and “post-occupancy evaluation”). 

One can view the ADDIE model as a recipe for instructional design. Novice cooks follow recipes, without 
modification or extrapolation, getting the expected result (cf. Ratatouille, Bird, 2007). Completion, or done, is 
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desired. For the great cook, however, we know that it is not how religiously they follow the recipe, but rather how 
they go beyond what is proscribed. Chefs, those we need to advance the field if not our culinary experience, make 
use of their imagination, based on a process, but also are not hindered by it. 

The process of design cannot be codified in a simple recipe in any sub-field, from graphic to urban design. 
Instructional design, although ostensibly following a single lockstep process, also exhibits this same diversity of 
method. In a study of the methods of instructional designers, Visscher-Voerman and Gustafson examined the design 
processes of instructional designers. Most followed a traditional, rational, ADDIE based model, but their research 
showed that "design processes are much more heterogeneous and diverse" than the ADDIE model suggests (2004). 

Logically various variations of ADDIE have been proposed, essentially slight deviations from the standard (see for 
example. Gibbons, Visscher-Voerman, Hoadley & Cox, Parrish, Wilson, Reeves). Unfortunately, within these 
proposals, the total process isn't changing, just a different visual analysis of the steps of design. These proposed new 
models inlcude the star model, waterfall model, and spiral model, which minor changes of the process; they are 
ADDIE with make up. 

The use of this proscribed method in teaching instructional design is understandable. Providing learners with a set of 
procedures to follow that is known to generate a predictable results is, to some extent, valuable. Novices in any field 
commonly seek a specific set of tasks that will guarantee success; however, as designers mature and address more 
complex problems, they generally decrease their use of a specific set of ordered actions. As noted in Dreyfus and 
Dreyfus (1986), novices in any field tend to seek rules and follow rule based behavior, while as an individual 
progresses in (design) skills, knowledge becomes less tacit, rules less explicit, and capability less defined by 
declared knowledge. 

"During the first stage of the acquisition of a new skill though instruction, the novice learns to 
recognize various objective facts and features relevant to the skill and acquires rules for 
determining actions based on those facts and features. Elements of the situation to be treated a 
relevant are so clear and objectively defined for the novice that they can be recognized without 
reference to the over all situation in which they occur. We call such elements 'context-free,' and 
the rules that are to be applies to these facts regardless of what else is happening 'context-free 
rules'. " Dreyfus & Dreyfus (1986) 

Mapped to the use of the ADDIE model, this anticipates that the higher the design skill of the instructional designer, 
the less use and less applicable is this model. Novice designers will use the model with fervor, evaluated on their 
loyal steps in the process, as (theoretically) consistent with design skill. Educators use the method as a way to codify 
the process and produce standardizable results. 

In reality, this model perpetuates a process of design that is lockstep in execution, boring to the designer, and 
frontloads the “good” parts of the design process. It is a disservice to the novice instructional designer to present this 
method as the sole process of instructional design. Later aspects of the process are demotivating; implementation is 
viewed as simply getting the job done and/or drudgery; evaluation is an afterthought if remembered. 

Design is, of course, never done; good designers have in their psyche an impatience, a dissatisfaction with the status 
quo. They seek the challenge, the unexpected result. The goal of design education is not to produce consistent 
designs, particularly those that replicate previous designs. The goal must be to produce better designers and hence, 
designs as yet unconceived. Inherent in this goal for the profession is one of constant improvement and innovation 
of the design process. 

How this is done is an ongoing debate in most design fields, with the exception of instructional design. Most design 
fields recognize the value of extensive work in design studios, addressing increasingly complex design projects. 
However, the design studio method of remains rare within instructional design education. Within the studio and 
within the design process is embedded the use and application of current theory, ideology, and professional 
behavior.. 

While being a designer implies having the explicit knowledge necessary, the implicit procedures, values, and 
attitudes needed to successfully design must also be explicitly developed. Design can be viewed as a system of 
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beliefs, with expected behaviors, skills, and aesthetics. These aspects will advance the field of instructional design 
much more than technical or theoretical expertise. 

Any design process has a series of different activities, with attention spent in various different aspects of the work. A 
broad understanding of the project, as well as background in the field and ability in each aspect of the work is 
needed. Some time must be dedicated to the experimental aspects of the work, both on specific projects and to 
generally advance. A rigorous understanding and evaluation of the field's body of knowledge must be applied. And 
the project must be implemented with skill and continuous improvement of the design, even after formal 
completion. 

A number of theories and ideas guide the practice of instructional design, but we are also guided by our perceptions 
of our own practice. If we view the work of instructional design as the application of theories of cognition and 
learning, the inherent value is one of the learning sciences or perception, and that will guide the work. Similarly, if 
we view the process as simply one of the development of instructional materials, we will create work that is 
complete but limited. We must seek to include the full range of roles of the designer in every project, extending our 
self-image beyond that of the scientist or engineer. 

One way to organize and present the values of instructional design is to use and emulate a series of roles as modes or 
exemplars for successful design practice. Being a designer, and acting as a designer, therefore, becomes more 
important than understanding what a designer does. Being able to be creative is much more important (and difficult) 
than knowing what creativity is. Presented here is a new design process, one which entails a number of "roles" 
instead of tasks. 

Each of these exemplars of the process could also be described in a procedural tone, outlining actions that need to be 
taken, artifacts that need to be produced. Each will, of course, be inherently structured by sub-tasks. The process is 
not task driven, however, a major change will only come through changes in values, belief, attitudes and perceptions 
through revising our own perceptions of our work. 



Roles 

Instead of a recipe for making instructional design, we present a series of models for the behavior of instructional 
designers. There are a series of roles that procedurally will lead one through instructional design, and, more 
importantly, also act as a series of exemplars, which if followed, will improve quality and innovation within the field 
of instructional designers. We seek to make better instructional designers, not by providing a low level list of 
actions, but by giving the field a series of models of quality in each aspect of design work. 

The roles of design we present here are archetypes, i.e., romanticized versions of real professions, exemplars of 
behavior and practice, which are valid as models for professional behavior in instructional design. As exemplars, we 
seek from them the best of their practices; for example, from the artist, creativity, and from the craftsman, patience 
and advancement through practice. 

Each of the roles we have selected is well known through our society: artist, architect, engineer, and craftsman. Each 
title is often said with pride, or bestowed on another as praise. For example, describing someone as an artist denotes 
a creative skill with a medium, while an engineer brings a logic and reason backed by scientific knowledge. The 
term "designer" can also serve as an exemplar, when the full diversity of the work of design is included. 

Each of the roles we highlight have been present in design for a long period of time, and most earlier design 
practices necessarily included all these attributes. In the Renaissance, these roles were blurred, integrated into the 
single individual or practice; for example, Leonardo da Vinci was artist and engineer, architect and craftsperson. 

The practice of instructional design currently focuses principally on two roles, which we describe as an instructional 
engineer, and the other as an instructional manufacturer. 

Much of the field seeks the scientific, rational approach to design, where answers exist, and the best method can be 
found, then adopted by all, and then developed through completion. 
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The Instructional Engineer 

We use the term "instructional engineer" as an aspect of instructional design that is most addressed in instructional 
design programs. The instructional engineer focuses on the applying research derived models for learning. It is close 
to our vision of scientist, someone seeking new knowledge through research, but here as the role is an applied one, 
the term "engineer" is most relevant. Indeed, some argue that the field of "instructional design" itself is mis-named: 
“Some object to the word ‘design,’ suggesting as it does a rather arty orientation, and insist that what we really need 
is ‘instructional engineering" (Shepard, 2002). 

Engineering is the creative application of scientific principles used to plan, build, direct, guide, manage, or work on 
systems to maintain and improve our daily lives. While scientists explore nature in order to discover general 
principles, engineers apply established principles drawn from mathematics and science in order to develop 
economical solutions to technical problems. 

In our model, the role of the instructional engineer is one of instructional problem solving. Most engineers, either in 
the instructional field or in the main fields of engineering such as civil, structural, or mechanical engineering are 
highly trained professionals. In education, the instructional engineer ensures a product is usable by the target 
audience and makes the product achieve its educational goals. Contemporary, research-based ideas are used to 
develop instructional materials; educational theory is an important component of the work of the instructional 
engineer. The principle goal of the engineer is the functional efficiency of the work, planning and organizing the 
project. These are valuable aspects of the design process and can advance the value of the work. 

In current practice, most design completed by the instructional engineer is passed on to technicians with little 
opportunity for change. In any design project, at some point, the conceptualization, the planning, the broader view 
have been completed, and the work must be implemented. Here too there are significant questions, of a choice 
between completion and craft. Most instructional design work these days is manufactured; ideas developed 
elsewhere are implemented by workers divorced from concept, aesthetics, or theory. 

The instructional manufacturer 

Most materials produced in the field of instructional design are completed by an manufacturer and not by an 
engineer. The manufacturer frequently is a technically skilled individual applying a pre-defined design template to 
solve an educational problem, delivering results as efficiently as possible. The solution to an educational problem is 
given or dictated to the manufacturer, whose responsibility ins one of simple, recipe formatted production 
efficiency. Production consistency and stability are of primary importance, resulting in products that are predicable 
and functional. As one expects a recipe from a cookbook to be predictably good but also what was intended, one 
should expect the results from a manufacturer to produce consistent, but not innovative work. 

For example, when asked to develop educational materials for use through distance education, the instructional 
manufacturer might employ traditional instructional design methods to develop instructional materials emphasizing 
content presentation and application. Such materials are commonly delivered to learners via the most efficient 
technologies (e.g., online quizzes, Blackboard/WebCT templates, PowerPoint presentations, etc.). Most of these 
technologies are stable and, at the core, are based on educational theories such as constructivism, collaboration, or 
cognitive science, but such theories are remote from the manufacturer. Models for the design process would focus 
on the functional (i.e. "form follows function"). As with the architecture in the 1960's, an aesthetic could develop 
based on making the c technology work, on utility. 

There are two major problems with this simplified process. First, design ceases with the conclusion of the 
engineering phase, and all prospects for qualitative improvement stop. The experience and technical skill that may 
be present with the manufacturer are seldom integrated with the scientifically based knowledge of the engineer. And 
second, the criteria for success of the manufacturer is one of quantity, not quality, of the number of functional 
widgets of pure production: Is it done? While efficiency may go up, quality does not increase. 

What are the values of the manufacturer? Speed, consistency, and completion of the assigned tasks. 

Contrary to the role of the manufacturer is one of the craftsperson. We seek to treat this phase of project 
development as critical to the worth of the end artifact; as part of the full design process. This phase is one which 
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adds value to the project. It is needed, for the health of the process and the participant designers, that this portion of 
the work be a positive, additive, and generative portion of the work. 

The Instructional Craftsperson 

The instructional craftsman encompasses the work of implementation, but also still seeks to improve the project or 
design. Traditionally, craft work implies a high level of skill in execution, and while not having a focus on the 
research or theoretical foundations. It still has a good theoretical understanding of the field. 

As a verb, 'to craft' seemingly means to participate skillfully in some small-scale process. This 
implies several things. First, it affirms that the results of involved work will still surpass the results 
of detached work. To craft is to care. Second, it suggests that partnerships with technology are 
better than autonomous technology. For example, personal mastery of open-ended software can 
take computers places that deterministic software code cannot. Third, to craft implies working at a 
personal scale— acting locally in reaction to anonymous, globalized, industrial production— hence 
its appeal in describing phenomena such as microbreweries. Finally, the usage of 'craft' as a verb 
evades the persistent stigma that has attached itself to the noun. (McCullough, 1998, 21) 

In modern society today, we have a view, a vision of "craftsperson", one of a highly skilled trades worker doing 
exceptional work, a benevolent artisan. Historically, a craftsperson was a highly skilled guild member, required to 
take in an apprentice to continue the guild. Inherent in the role of master craftsperson was the requirement of 
building the work and the next generation. 

One can imagine a craftsperson building a a boat or wood strip canoe, by hand. The work is comparable to 
manufacturered efforts, but while similar, it does not regress to the level of detached reproduction by a human. The 
maker, the individual is engaged with the work. To some extent, the craftsperson is somewhat isolated from 
concerns of reality; in their own time, patient and still efficient, the work, not the schedule is of prime importance. 
It's done when it's done. 

Our vision the instructional craftsperson includeds a high level of implicit knowledge developed from experience. 
They seek quality in both technical and aesthetic terms. They value the product more than the user or client; we 
expect physical manifestations of their work in their lives; calluses and patience. 

Most practitioners today would view the addition to or description of the final phase of their work as an easy change 
to the process of instructional design. Adding the title "craftsperson" to the completion phase of the work does not, 
however change the process. This could be window dressing, and would amount to little more than lipstick on a pig. 
If the craftsperson is an appendage, a renaming of the manufacturing phase, there will be no true improvement, and 
will have the same real impact as calling in the graphic artist to apply some aesthetics to the project. For there to be 
value in craft, it must have a voice throughout the design, and be a real value in execution. 

The building craftsperson, the mason of the Renaissance, evolved through time to become the architect of today, as 
buildings are now designed. There are still masons today, focusing more on production, but as design has become 
more complex, and as design has separated from construction, the role of architect has evolved as separate. 

Architecture is a profession that still values craft and still seeks to train new architects in the means of production. It 
also educates practionners in the results of research and values aesthetics and craft. This holistic view, a broader 
approach to the design process is applicable too to the field of instructional design. 

Architects today are expected to integrate all the functions of the design process, from initial divergent 
conceptualization to final evaluation. Architects are current with the newest research and technologies, have the 
skills to work in a variety of media, and integrate the needs of the user and the client with design ideas. We view this 
phase/role as one which includes a broad range of responsibilities; the instructional architect holds a viewpoint that 
is holistic, looking across the entire project, not just within the current activities. 
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The Instructional Architect 



We view the role of instructional architect as one that has a balanced approach to instruction design, one which 
values aesthetics and innovation, applied current research, and which critically examines the solution to increase 
user engagement, motivation, and interaction. Instructional architects are unique in that they are not satisfied by 
simply solving the problem; the architect is motivated by extending the boundaries of the resources to explore 
solutions that enhance learner experience, moving beyond the educational and technological specifications of the 
instructional problem (i.e. design beyond done). 

Developing an understanding of the entire project is critical to the design process; one must have a holistic view of 
the design challenge. Inherent in this understanding is an identification and recognition of the assumptions of the 
design problem, and a questioning of the design problem itself; what is the true nature of this design problem? This 
phase also examines the resources at hand and the theoretical and philosophical orientation of the project. 

The instructional architect extends the engineer’s functional and usable solution and attempts to incorporate 
aesthetics at the core of the design process. By doing so, the architect explores divergent solutions that extend and 
cultivate the affordances of a medium. The architect’s approach to instructional design attempts to balance utility, 
usability, and aesthetics. 

Having a broad viewpoint in the process of design is essential to a successful project, but beyond that wide view, the 
designer needs to specifically address the development of new ideas. In most design projects, a single driving 
concept is selected very early in the process. These ideas are generally pre-conceptions, ideas of what works and 
what could easily be done, and sadly, they are also ideas that have already been successfully executed. In order for 
new ideas to be adopted or even be conceived, the successful designer needs to explore many ideas; ideas that are 
different, unusual, that will fail or that will break the mold. In short, the instructional designer must work as an artist. 

The Instructional Artist - 

The final exemplar is that of the instructional artist, an iconoclastic divergence that embraces experiment and failure. 
Here the process of instructional design examines ideas that don’t work, paths that are not expected, and allows for a 
more diverse range of conceptualization. Within the field of creativity training, there are a number of techniques that 
encourage examining wrong answers or the opposites of the expected results. Similarly, the phase of the 
instructional artist is fraught with failure, and one which diversifies thought. Here is where most innovation in the 
field will occur, not in the later roles of engineer or manufacturer, where 1% improvements are accepted as goal. 

The wager of the artist is to win big, with substantial increases in the value of designs, understanding concurrent 
risk. 

Here, in this writing, we view artists as those with a mastery of a medium, with an intense focus on their work and a 
concern for aesthetics. They exhibit a high level of creativity, even to the point of working outside of society. 

Failure, unexpected results, and disturbance of the status quo mark the work of the artist; advancement of the finish 
product is not necessarily the goal, but is rather an advancement in the understanding and development of new ideas. 

We view the role of the artist as one providing divergent thinking at the beginning of any project; providing 
aesthetic direction and inspiration throughout the project, and acting as the "what if' person on a project team. 

The artist is an instructional explorer. The artist uses instructional problems as stimuli to experiment with media and 
affordances. The instructional artist may work without client or audience, only later attempting to apply to 
instructional practice what has been learned through the artistic experience. The artist embraces failure and engages 
in continuous self-criticism while attempting to understand both the problem and self. 

Each role is critical at some point in the process, from the creativity of the artist, to the care and completion of the 
craftsperson; each serves as check and balance for the other roles, the engineer bringing the artist back to earth, the 
architect reminding the craftsperson of the needs of the client; and each is constant and integrated into the entire 
process, not taking the lead all the time, but present and engaged throughout. 
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Roles as process 

We view these roles as generally sequential. Each role, in turn, leads the project, applying their own expertise: 

Artist, Architect, Engineer, and Craftsperson. This sequence is, of course, similar to many other iterations of design 
process, but given the use of these roles, these exemplars, each phase has it's own values and quality. 

As with many other things, the methods and products of instructional design represent the values of the designer; our 
arguments here may be ones of belief. Present today in the instructional field is a belief that design is a purely 
rational and logical solution of problems, or a belief that inherent in any design must be aesthetic, spiritual, and 
philosophical aspects, or a belief that design must be inclusive, and spring from the ideas and actions of the learners. 

Each of these roles may be performed as part of an individuals work on a design project, or they can be assigned to 
different parties of a team effort. They can be followed in sequence, but it must be understood that each aspect, each 
exemplar participates throughout the design process. For example, during the engineer phase, the sensibilities of the 
artist must still be present. 

The Role Based Design Process can be used to organize and manage large teams or it can be used for projects 
designed by small teams or individuals. 

Critical to the success of the process is the integration of the four design roles in the process; that artist, architect, 
engineer, and craftsperson are present throughout the entire sequence, although each role will take a lead in the 
design of a project. 

Role based design process can be of value for instructional designers of all levels. For the experienced designer, a 
procedure for design is often in place. Applying new models or roles will help change the outlook and results. The 
use of a role based process will remind the experienced designer of other, divergent aspects of design methods, and 
serve to stimulate directed reflection as part of the process. The checklist is not used, but rather an understanding of 
different components of a complete designing experience. 

For the beginner, use of a formalized linear design process can lead one through a challenging sequence of 
procedures; as artist, architect, engineer, and craftsperson; a checklist of tasks can help understand the process as 
well. Using Role Based Design in lieu of ADDIE will encourage an inexperienced designer to include aesthetic 
components through out the design process, to view the entire process as a whole; and to be encouraged to innovate 
as opposed to replicate design models. 



Discussion 

“You don ’t take a photograph, you make it. ’’Ansel Adams (1902-1984) 

Ansel Adams was lauded by many as the pioneering and visionary black-and-white landscape photographer of the 
20 th century. Throughout his life Adams exemplified the role of an artist by combining art, technology, spirituality, 
and an adventurous value for failure to capture and share visions of the American West through careful and 
conscious creation. Fie epitomized the values, philosophies, and practices of the Artist, Architect, Engineer, and 
Craftsperson roles; he is presented here as an example of Role-Based Design. 

Adams is renowned as an artist for his stunning work, and we know that preceding that are years of experiment, 
trial, failure and exploration. Fie carried that vision of exploration without reward for many years, literally 
wandering in the wilderness. 

Acting as the holistic architect, Adams developed the ideal of previsualization, a process of visually and 
conceptually exploring a scene and seeing in the mind’s eye the final photographic print before film is exposed. 
Lacking this exhaustive phase of holistic conceptualization, Adams believed resulting photographs would only be a 
product of inspired luck at best, and, at worst, shallow and unable to communicate meaning. 

With a visual objective in mind, Adams, acting as an engineer, used his scientific understanding of a composition’s 
tonal values to capture the vision on photographic glass. After years of meticulous conceptual and technical 
refinement, Adams created the Zone System, a systematic approach of precisely defining the relationship between 
the visualized photograph and the final result, to ensure all light and dark values of a scene render effectively onto 
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film. The Zone System has been practiced and embraced by thousands of photographers, solidifying Adams as both 
an inventor and educator. 

Adams, a true craftsperson, is often best acknowledged for his unprecedented mastery in the darkroom. With the 
photograph captured on film, many photographers view the darkroom, as solely the production of a captured image 
on paper. For Adams, however, the act of creation did not stop when the picture was taken; the darkroom was where 
the photograph is made. 



Embracing Role-Based Design 

In lieu of an habitual, rearticulated summary of the Artist, Architect, Engineer, and Craftsperson roles, we will sever 
scholarly tradition and conclude with a collection of 12 questions one can reflect upon before, during, and after each 
design project. The premise of Role-Based Design is illustrated in people, not steps or processes; Role-Based Design 
encompasses the values, mindsets, philosophies, characteristics, responsibilities, traditions, and practices of real 
designers. 

Artist (playful experimentation) 

• When listening to the initial problem, how did I freely explore a variety of aesthetic, technological, and 
pedagogical possibilities (rather than applying past design solutions to the current obstacle)? 

• What are some of the creative, unique, simplistic, complex, innovative, and bizarre ideas I exhausted when 
exploring the problem? 

• In what ways have 1 failed during my design experimentation? 

Architect (holistic conceptualization) 

• What are the pedagogical, technological, and aesthetic characteristics/affordances of the proposed solution? 

• How does the conceptualized solution provide opportunities for transformation in learning and/or 
instruction? 

• What steps have I taken to create an instructional experience for the learner, as opposed to an instructional 
product? 

Engineer (scientific realization) 

• What are the physical, logical, pedagogical, technological, and cultural constraints of the design and 
implementation? 

• What structural and technical features have I implemented to ensure scalability and sustainability of the 
solution over time? 

• What measures have I taken to ensure a reliable, valid, and pedagogically-sound solution? 

Craftsperson (experienced evolution) 

• Have I improved upon the design conceptualized by the architect, or have I simply developed the final 
product to specification? 

• How have I affected the quality of ideas, processes, and production? What are 6 things I could have done 
better during this project (i.e. conceptual items, procedural items, and developmental items)? 

• What have 1 learned from this project that will ensure a higher quality of design and user experience for my 
next project? 
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Abstract 

The demand for good instructional environments presupposes valid and reliable analytical instruments for 
educational research. This paper introduces the SMD Technology > (Surface, Matching, Deep Structure), which 
measures relational, structural, and semantic levels of graphical representations and concept maps. The reliability 
and validity of the computer-based and automated SMD Technology’ was tested in three experimental studies with 
106 participants. The findings indicate a high reliability and validity. The discussion focuses on the development 
and realization of the three levels of the SMD Technology and applications for research, learning and instruction. 



Introduction 

The demand for good instructional environments presupposes valid and reliable tools, instruments and 
methodologies for educational research. However, many of them are developed with little or no theoretical 
justification, which leads to doubtful findings and no contribution to the improvement of learning environments 
(Novak, 1998). Accordingly, the development of new tools, instmments and methodologies to capture key latent 
variables associated with human learning and cognition requires a solid theoretical foundation. 

One central interest of psychological and educational research is internal cognitive processes and systems, 
which are described by theoretical constructs such as mental models and schemata (see Seel, 1991). However, 
mental models and schemata are theoretical scientific constructs which are not directly observable. Accordingly, 
researchers can only learn about mental models or schemata if (1) individuals communicate their internal systems 
(Seel, 1991) and if (2) valid and reliable instmments and methodologies are used to analyze them (Seel, 1999). A 
wide variety of empirical approaches for the analysis of external representations of mental models and schemata 
exist (see Al-Diban, 2002), but they often lack a solid theoretical foundation and their analysis is considered to be 
very time consuming (Ifenthaler, 2006). On the other hand, new technologies such as concept mapping tools are 
being introduced into learning environments, but the analysis of data collected with such new technologies still 
places a huge demand on methodologies. 

The purpose of this paper is to introduce the computer-based and automated SMD Technology’ for 
relational, structural, and semantic analysis of graphical representations and concept maps. We first introduce the 
theoretical constructs of mental models and schemata as a key concept for understanding human learning and 
problem solving processes. Second, the complex processes of externalizing internal knowledge representations (re- 
representation) will be discussed. Third, we introduce our own SMD Technology’, which enables us to measure 
graphical representations and concept maps with three different quantitative indices. We then focus on the empirical 
reliability and validity testing of the SMD Technology’. Finally, we introduce a broad field of applications for the 
SMD Technology within the field of research, learning, and instruction. The article ends with a conclusion and 
further perspectives. 



Background 

Mental models and schemata are theoretical constructs for understanding human learning and problem 
solving processes. Following the verdict of Piaget (1950, 1976), we argue that new information is processed by the 
complimentary processes of assimilation and accommodation. According to Seel (1991), a person can assimilate 
new information as long as an adequate schema can be activated. If the activated schema does not match exactly, it 
can be adjusted by means of accretion, tuning, or reorganization. The accretion process is defined as an 
accumulation of new information to the existing schema. Tuning can be described as a change of single components 
within the activated schema. The result of a successful adjustment of a schema is a subjective plausible solution of a 
problem or the understanding of new information. However, if the processes of accretion and tuning are not 
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successful or if no schema is available at all, new information can only be accommodated by the process of 
reorganization. According to Seel (1991), the process of reorganization is realized by constructing a mental model 
(see Figure 1). 




Figure 1: The process of assimilation and accommodation 



Mental models are dynamic ad hoc constructions of individuals that provide subjective plausible 
explanations on the basis of restricted domain-specific information. Johnson-Laird (1983) describes the model 
building process as a step-by-step reconstruction of an initial mental model (fleshing out). Additionally, the 
reduction to absurdity (Seel, 1991) is used to test whether the activated mental model can be replaced by another 
mental model. Flowever, as long as an activated mental model provides enough subjective plausibility to meet the 
requirements of a phenomenon to be explained, there is no need for the construction of a new mental model. Seel 
(1991) assigns mental models four general functions, (1) simplification, (2) envisioning, (3) analogical reasoning, 
and (4) mental simulation. Depending on the objective of the model-building person, one of the four functions is 
used for the mental model building process. In comparison to the activation of an available schema, the mental effort 
for the construction of a mental model is higher and more time consuming (Seel, 2007). 

Accordingly, learning, reasoning, and problem solving involve the construction of mental models and 
schemata. In order to support successful learning, reasoning, and problem solving, it is necessary to investigate the 
mental model building process precisely. However, as it is not possible to measure internal representations of 
knowledge directly (e.g. schemata, mental models), the following paragraph will focus on the complex processes of 
externalizing internal knowledge representations. 

Externalization of Internal Knowledge Structures 

Theoretical constructs such as the mental models and schemata discussed above are used by cognitive and 
educational researchers to explain the complex phenomenon of human learning, reasoning, and problem solving. As 
long as these internal knowledge structures are not directly observable, researchers require adequate tools, 
instruments, and methodologies to allow people to externalize them. The process of externalization is considered as 
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a conscious process of communicating mental models or schemata using adequate sign and symbol systems (see Le 
Ny, 1993; Ifenthaler, 2006). Hence, externalization can be realized through speaking out aloud, writing a text, 
drawing a picture, or constructing a diagram, graphic, or concept map (see Hanke, 2006). 
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Figure 2: Interrelation of internal and external representations 



As shown in Figure 2, we are able to distinguish between internal representations (e.g. mental models, 
schemata) and external re -representations (communicated using adequate sign and symbol systems). Furthermore, 
we argue that these two types of model representations are interrelated. First, through the process of internalization, 
a person is able to construct a mental model or activate an available schema. From the point of view of instructional 
design, the process of internalization is where we can systematically influence the construction of mental models by 
providing well-designed external re-representations (e.g. learning materials, feedback, etc.) of phenomena to be 
explained (e.g. Norman, 1983). 

Second, the process of externalization enables a person to communicate his or her understanding of 
phenomena in the world. This perspective is the only way in which researches can learn more about a person’s 
internal representations. Accordingly, adequate tools, instruments, and methodologies for the analysis of mental 
models or schemata can only be developed with a clear understanding of the complex processes of internalization 
and externalization. Although it appears to be possible to assess internal representations through their externalized 
re-representations, we need to keep in mind that the re-representations might be biased through the lack of 
communication skills, the use of inadequate sign and symbol systems or the use of insufficient research instruments. 

Therefore we argue that instruments used for the analysis of such constructs must have a strong theoretical 
foundation and be tested for reliability and validity (Seel, 1999; Ifenthaler & Seel, 2005). A detailed review of 
methodologies for the assessment of graphical representations revealed a huge demand for an automated and 
computer-based tool (see Ifenthaler, 2006). As a result, we developed our own SMD Technology. 



SMD Technology 

Based on the theory of mental models (Johnson-Laird, 1983; Seel, 1991) and graph theory (Harary, 1974; 
Chartrand, 1977; Bonato, 1990; Tittman, 2003), the computer-based and automated SMD Technology (Surface, 
Matching, Deep Structure) uses (a) graphical representations such as concept maps or (b) natural language 
expressions to analyze individual processes in persons solving complex problems at single time points or multiple 
intervals over time. In the following, we define the externalized knowledge structures as a model M. 



animal cells 




plant cells 

Figure 3: Model M 3 composed of two propositions P, 
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Depending on the elicitation process (e.g. using the Structure Formation Technique [paper and pencil]; 
concept mapping tools [computer-based]; natural language statements [computer-based or paper and pencil]), the 
raw data should be stored pairwise (as propositions P,) including (a) the modelnumber as an indicator of which 
model a proposition belongs to, (b) nodel as the first node of the proposition, (c) node2, which is connected to the 
first node, and (d) a link which describes the link between the two nodes (see Figure 3 and Table 1). 



Table 1 : Raw data of a model stored pairwise (as propositions) 



modelnumber 


nodel 


node2 


: link 


003 


cells 


animal cells 


consists of 


003 


cells 


plant cells 


consists of 









After the raw data has been transformed into the standardized format (see Table 1), it is stored on a SQL 
(structured query language) database. Flowever, the transformation process of paper and pencil models (e.g. 
Structure Formation Technique) is very time consuming. Therefore, we recommend the use of computer-based 
elicitation techniques which already support the standardized format (e.g. DEEP; CMap; MITOCAR) in order to 
guarantee a more economical analysis and additionally a highly reliable transformation process (see Ifenthaler, 
2006). 
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analyze 



clear | 



Figure 4: User interface of the SMD Technology 

The automated analysis process of the SMD Technology will be started by the researcher through the User 
Interface , where all stored models in the SQL database can be selected (see Figure 4). After selecting the models M, 
for the analysis process, the system will automatically calculate three numerical indicators - Surface, Matching, and 
Deep Structure - and generate standardized graphical re-representations for each individual model M, (Ifenthaler, 
2006). 
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Surface Structure 

The relational structure of each individual model M-, is represented on the Surface Structure. This simple 
and easily calculable indicator is computed as the sum of all propositions P, in a model M-,. 

0=i.p, ini 

1=0 

0 is defined as a value between 0 (no proposition = no model) and n (n propositions P t of a model Mf The 
Surface Structure of model M 3 , represented in Figure 3, would result in 0 = 2. According to the theory of mental 
models (Seel, 1991), the number of nodes and links or propositions a person uses is a key indicator for the 
investigation of the progression of knowledge over time in the course of problem solving processes (see Scandura, 
1988). However, although this first indicator enables a rapid and economical analysis of the relational structure of a 
model Mi, additional indicators are required for a more detailed analysis. 

Matching Structure 

The structural property of a model M t is displayed on the Matching Structure. The second level of the SMD 
Technology > indicates the range and complexity of a model M-,. 

fl = max{d(i, j)} [1.2] 



(i is computed as the diameter of the spanning tree of a model M, and can lie between 0 (no links) and n. In 
accordance with graph theory, every model M, contains a spanning tree. Spanning trees include all nodes of a model 
M, and are acyclic (see Harary, 1974; Tittman, 2003). Figure 5 illustrates model M 5 and its corresponding spanning 
tree. 
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Figure 5: Model M s and its corresponding spanning tree 

A diameter is defined as the quantity of links of the shortest path between the most distant nodes. For the 
calculation of the Matching Structure index, the spanning tree is transformed into a distance matrix D. 
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The Matching Structure index is calculated as the maximum value of all entries in the distance matrix D. 
The diameter or Matching Structure of the spanning tree in Figure 5 is calculated as follows: 

fl = max {J (/,/')} = 4 [ 1>4 ] 

The change in range or complexity of a person’s model M, is our second key indicator for the analysis of 
learning and problem solving processes (see Seel, 1991; Ifenthaler, 2006). 

Deep Structure 

The semantic composition of a model M, is measured on the Deep Structure. The Deep Structure is 
calculated with the help of the similarity measure (Tversky, 1977) as the semantic similarity between an individual 
model Mj and a reference model M r . A reference model M, is defined as a subject domain-specific model (e.g. expert 
solution; another subject’s model; the same subject’s model constructed at a different time point). 

In contrast to the graph theory-based calculation of the Surface and Matching Structure, model analysis on 
the Deep Structure is realized through a similarity calculation between a model Mj and a domain-dependent 
reference model M r . Hence, a reference model M r of high quality is a necessary precondition for a comprehensive 
analysis of the Deep Structure. 

A similarity measure describes the degree of similarity between two objects, represented by a number 
between 0 and 1. Decisive for a similarity measure are objects with similar and different features. Tversky (1977) 
considered an object as an amount of features. The identification of a similarity between two objects is realized 
through a comparison of their features. The similarity formula takes not only the amount of similar features into 
account, but also the amount of different features. Lin (1998) defines similarity with the following three statements: 

• The similarity between A and B is related to their commonality. The more commonality they share, 
the more similar they are. 

• The similarity between A and B is related to the differences between them. The more differences 
they have, the less similar they are. 

• The maximum similarity between A and B is reached when A and B are identical, no matter how 
much commonality they share. 

Accordingly, the smallest similarity between two objects A and B is given if no common features exist. In 
this case, the two objects are completely different and the similarity measure is 0. The similarity measure increases 
with a rise in the number of common features. A complete similarity of all features results in a similarity measure of 
1. 

The similarity of models on the Deep Structure is identified through the feature „proposition“ - the 
semantic characteristic of the proposition. The Deep Structure index 5 is defined as the Tversky similarity between a 
model Mj and a reference model M r . In general, we calculate: 

s = fiAnB) 

f(AnB) + a-f(A-B) + 1 6-f{B-A) [L5] 



A and B are the amount of propositions of a model comparison. The function f(M) corresponds to the 
number of elements in the amount M. The parameters a and ft control the weighting of similar and different features. 
Both similar and different features are considered in the calculation if the weighting of a and ft is equal (a = fi = 0.5). 
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The value of the Deep Structure index 8 is defined between 0 (no semantic similarity between the models) and 1 
(absolute similarity between the models). 
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Figure 6: Model M 6 and reference model M r 

The Deep Structure or semantic similarity between model M 6 and reference model M r is calculated in an 
automated iterative process. Every proposition in model M 6 is analysed for similarity with every proposition in the 
reference model M r . The Deep Structure index is calculated as follows: 

8 = 0.57 [ 1 . 6 ] 

Thus, the semantic similarity between model M 6 and reference model M r is 5 = 0.57 or 57%. The 
quantitative measures of the Surface, Matching, and Deep Structure can be used for further statistical analysis. A 
qualitative analysis is made possible with the standardized re-representations of the SMD Technology. 

Standardized Re-Representations 

The standardized graphical re -representation of the subject’s data is constructed as an undirected or 
directed graph with named nodes and links. This automated feature of the SMD Technology is realized with the help 
of the open source graph visualization software GraphViz (Ellson et ah, 2003). For every single analysis, four 
standardized PNG (Portable Network Graphics) images are generated. Images (1) and (2) are the re-representations 
of model M, and reference model M r (for an example see Figure 6). Image (3) represents the similarity model, 
including only the nodes and links which are semantically similar between model M t and reference model M r (see 
Figure 7). 
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Figure 7: Similarity re-representation of model M 6 and reference model M r 

Image (4) is defined as the contrast model. It includes only nodes and links which have no semantic 
similarity within model M, and reference model M r (see Figure 8). 
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Figure 8: Contrast re-representation of model M 6 and reference model M r 

Experimental Validation of the SMD Technology 

To investigate the objectivity, reliability, and validity of the computer-based and automated SMD 
Technology’, we conducted three quasi-experimental studies. The objectivity of the SMD Technology’ was guaranteed 
by the computer-based and automated realization of the instrument. In the following section we report our results for 
reliability and validity of the SMD Technology. 

Subjects 



Three quasi-experimental studies (Studies 1, 2, and 3) were conducted with 106 subjects (70 female and 36 
male) at the University of Freiburg. Their mean age was 18.3 years ( SD = 4.6). The subject domain of Study 1 was 
geology and that of Studies 2 and 3 was geophysics. The subjects spent five hours on successive days working on 
complex problems with a multimedia discovery-learning environment. 

Learning Environment 

The multimedia discovery-learning environment consisted of four modules. The modules could be divided 
into declarative and heuristic modules. The declarative modules contained all information needed to solve the 
phenomenon in question, while the heuristic modules primarily supported the model building process (see Dummer 
& Ifenthaler, 2005). 

Starting from the problem & learning task area, the subjects solve complex tasks from specific subject 
domains (Study 1: geology; Studies 2 and 3: geophysics). The subjects can navigate through different topics of the 
subject domain within the curriculum module. Additional information about the subject domain is provided in the 
form of various text documents, pictures, and audio recordings in the knowledge archive. The Model Building Kit 
(MoBuKi) provides the subjects with information about models, model building, and analogical reasoning. It 
contains three levels of abstraction of the material provided: (1) knowledge level; (2) procedural level; and (3) 
examples level. The toolbox is used to elicit the subjects’ understanding of the phenomenon in question. 

Procedure 

The three quasi-experiments took place in the computer laboratory at the University of Freiburg. Subjects 
had to solve a complex problem while working with a multimedia discovery-learning environment. The problem 
solution had to be elicited on six subsequent measurement points as a concept map. Every subject was given an 
introduction to the use and construction of concept maps. 

All subjects were randomly assigned to three types of treatments (see Ifenthaler, 2006). The groups were 
distributed as (a) scaffolding-based learning, (b) self-guided learning, and (c) control group. The subjects in group 
(a) received detailed feedback concerning their concept map during the model building process, subjects in group (b) 
received no feedback, and subjects in group (c) received no feedback and worked within a multimedia discovery- 
learning environment whose content was not linked to the complex problem to be solved. The quasi-experimental 
procedure consisted of three main parts: 
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1. Pretest Before the subjects were able to access the multimedia discovery-learning environment, a 
pretest was conducted which included: (a) the domain specific knowledge test; (b) elicitation of 
the preconception of the complex problem to be solved as a concept map; (c) a test on cognitive 
learning strategies (LIST-Test); (d) a test on intellectual abilities (BIS-Test). 

2. Model building process'. During the quasi-experimental session, the subjects were asked to solve a 
complex problem while working within the multimedia discovery-learning environment. At five 
measurement points, the subjects had to elicit their understanding of the complex problem in 
question as a concept map. 

3. Posttest'. The individual learning outputs were captured with: (a) a domain specific declarative 
knowledge test; (b) elicitation of the final solution to the complex problem as a concept map. 

The primary interest of the empirical investigation in this article is the experimental validation of the SMD 
Technology’. Therefore, we focus in the following section on reliability and validity tests. However, details on the 
learning-dependent progression of externalised models and treatment effects during the three quasi-experiments are 
reported in detail by Ifenthaler (2006) and Ifenthaler, Pirnay-Dummer, and Seel (2007). 

Reliability Test 

For the computation of the test-retest reliability (Spearman’s rank correlation), the Surface, Matching, and 
Deep Structure indices of measurement points three and four (control group) were used. 



Table 2: 



Test-Retest Reliability of the SMD 
Technology’ 





Test-Retest 




Reliability 


Surface Structure 


.824** 


Matching Structure 


.815** 


Deep Structure 


.901** 



** p < .01 (two-sided significance) 



The results in Table 2 show a high significant correlation between the indices ( Surface , Matching, and 
Deep Structure). Accordingly, this result is a broad hint for the reliability of the quasi-experimental study. On the 
other hand, we want to point out that mental models are individual ad hoc constructions (see Seel, 1991), and 
therefore standard reliability tests, e.g. Test-Retest-, Split-Half- or Odd-Even-Method (see Rost, 2005), have only 
limited validity as they consider the latent variable to be stable. However, the detailed research design of the three 
quasi-experimental studies and the applied learning environment guarantee at least an exact repeatability of the 
experiments. 

Validity Test 

Especially with newly designed and developed instalments (e.g. SMD Technology’), it is necessary to map 
theory based characteristics to measurable criteria. The goal of the construct validation is to determine from a 
theoretical point of view what the instalment really measures. For this purpose, several methodological best 
practices 1 are available (see Lienert & Raatz, 1994, p. 226). A comprehensive analysis of the theory of mental 
models (Johnson-Laird, 1983; Seel, 1991) and available instalments for the assessment of models constitutes the 



1 Coaelation of a test with several outside criteria; Correlation with tests with similar validation requirements; 
correlation with tests that assess other criteria; analysis of inter- and intraindividual differences in test results; 
factorial analysis (see Lienert & Raatz, 1994). 
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basis for the theory-based development of the SMD Technology’. From an empirical point of view, the validity of the 
SMD Technology is identified with the outside criterion (1) MITOCAR, and (2) declarative knowledge. 

Pirnay-Dummer (2006) developed the instalment MITOCAR (Model Inspection Trace Of Concepts And 
Relations), which enables a structural and conceptual analysis of natural language expressions. The raw data of the 
third quasi-experimental study (N=47) was analysed with the MITOCAR software. In the following, we use the 
results of the MITOCAR analysis for validity tests of the SMD Technology’. 



Table 3: Correlation between the SMD Technology and MITOCAR (N = 47) 





MITOCAR (concept and 
structure) 


Surface Structure 


Matching Structure 


MITOCAR (concept and 
structure) 


- 


.610**' 


.527** 1 


Surface Structure 




- 


. 166 **' 


Matching Structure 






- 



** p < .01; * p < .05 (two-sided significance) 
1 Pearson’s Correlation 



The results in Table 3 show significant correlations between the outside criterion MITOCAR and the 
Surface and Matching Structure of the SMD Technology’ 2 . After verifying convergent validity of the SMD 
Technology’, we want to test the SMD Technology’ with another outside criterion. This second validity test is for 
divergent validity on the basis of declarative knowledge. We assume that there is no correlation between the Surface 
and Matching Structure of the SMD Technology’ and the declarative knowledge measure. Further, we assume a 
correlation between the Deep Structure and the declarative knowledge. 



Table 4: 


Correlation between the SMD Technology and the declarative knowledge test (N = 47) 




declarative 

knowledge 


Surface Structure Matching Structure 


Deep Structure 


declarative 

knowledge 


- 


.273 1 .1 12‘ 


,355* 2 


Surface Structure 




.766** 1 


.089 2 


Matching Structure 




- 


. 1 66 2 



Deep Structure 



** p < .01; * p < .05 (two-sided significance) 

1 Pearson’s Correlation; 2 Spearman’s Correlation 

The results in Table 4 show no correlations between the declarative knowledge and the Surface and 
Matching Structure. This is consistent with the theoretical and methodological assumptions of the SMD Technology’ 
- the indices of the Surface and Matching Structure have no direct connection to the subject domain. The significant 
correlation between the declarative knowledge and the Deep Structure confirms the assumptions of the SMD 
Technology’ - we assume that persons with high declarative knowledge in a specific subject domain will also have a 
high Deep Structure index 8. To sum up, the empirical analysis revealed convergent and divergent validity with 
regard to the outside criterion. Additionally, the SMD Technology’ was part of a series of comparative studies of 
different quantitative and qualitative methodologies conducted in order to determine the methodologies’ strength 
and unique characteristics and to report collective validity (see Johnson et al., 2006). 



2 The Deep Structure index 8 of the SMD Technology compares the semantic similarity between a model and a reference model. 
This feature is not available with MITOCAR. Accordingly, the calculation of correlations between the Deep Structure and the 
MITOCAR indices is not necessary. 
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Applications for Research, Learning, and Instruction 



The use of different computer-based tools for re-representing knowledge structures (e.g. concept mapping 
software) has become increasingly accepted for research, learning, and instruction (Jonassen et al., 1997). In various 
research projects, concept maps have been used for analyzing learning outcomes, learners’ knowledge structures, 
and for self-assessment (see Mansfield & Happs, 1991; Eckert, 2000; Stracke, 2004). In the field of learning and 
instruction, concept maps have been used for providing feedback and advance organizers and for facilitating 
problem solving tasks (see Jonassen et al., 1997; Stoyanova & Kommers, 2002; Al-Diban, 2002; Ifenthaler, 2006). 
However, a large number of the available tools do not support automated feedback and analysis features. 
Accordingly, the development of the computer-based and automated SMD Technology’ opens up a broad field of 
applications for research, learning, and instruction. 

SMD Technology & Research 

Re-representations of knowledge structures are often analyzed by raters using diverse scoring approaches 
(see Taricani & Clariana, 2006; Jonassen et al., 1997). Depending on the research question, the raters focus on the 
quantity and quality of nodes and links, causal relationships, semantic content, direction and strength of links, 
hierarchy, or other visual arrangements. However, measuring the diverse information of individual concept maps by 
hand is very time consuming, and almost impossible for larger sets of data. Additionally, to guarantee high 
reliability and validity, every human rater must be an expert in the subject domain in question and in the application 
of quantitative and qualitative assessment strategies (Taricani & Clariana, 2006). Therefore, the automated analysis 
procedure of the SMD Technology’ calculates quantitative indicators of concept maps, which then can be used for 
further statistical computations. 

So far, the SMD Technology’ has been applied in different fields of mental model research. Ifenthaler (2006) 
investigated the trajectory of mental models constructed by subjects working on complex problem solving tasks. An 
HLM analysis of three quasi-experimental studies (N = 106) showed a significant increase of propositions when 
subjects worked for five hours in a multimedia learning environment ( Surface Structure). Accordingly, as long as 
new information is subjective plausible it will be added to a person’s knowledge structure. Further results indicate a 
significant increase in the diameter of the externalized knowledge structures {Matching Structure). Consequently, 
we found not only a significant learning-dependent increase in the number of propositions, but also a significant 
learning-dependent increase in structural complexity. 

In order to investigate the learning-dependent progression of novices’ mental models to more expert-like 
models, Ifenthaler (2006) compared the semantic similarity of externalized knowledge structures of novices with 
expert knowledge structures in different subject domains. The results of the Deep Structure indicator of the SMD 
Technology’ revealed a significant increase in similarity between novice and expert models. However, further HLM 
analysis indicated that the learning time of five hours was not long enough to integrate all information provided and 
consequently to gain higher similarity to an expert’s solution of a problem. Additionally, the provided learning 
materials and feedback could be improved for further experiments. 

Ifenthaler et al. (2007) investigated the role of cognitive learning strategies and intellectual abilities in 
mental model building processes using the Deep Structure indicator of the SMD Technology’. The results indicate 
that the training of mental model building skills is a complex problem which should be investigated further with 
regard to the roles of conditions based on the theory of mental models (Seel, 1991). 

Additionally, the SMD Technology has been used to investigate sharedness among team members (see 
Johnson et al., 2006). The focus on individually constructed concept maps and team re-representations can help to 
identify problems of team performance and lead to a better understanding of the complex performance processes 
within teams. Thanks to the flexibility of the SMD Technology, other indicators can be easily implemented in order 
to produce specific measures for a large number of research questions. 

SMD Technology & Learning and Instruction 

In the following, we will focus on the application of the SMD Technology’ for knowledge diagnosis, self 
assessment, and knowledge management. Other applications in the field of learning and instruction, such as analysis 
of navigation paths in learning environments (see Dummer & Ifenthaler, 2005), could be discussed on another 
occasion. 

In order to provide learners with the best possible learning materials, the instructor or an Intelligent 
Tutoring System (ITS) must be aware of their state of knowledge. In general, knowledge diagnosis is applied by 
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collecting necessary information about the learner with the help of various tests. By integrating the SMD Technology 
or parts of it (graphical re-representation; quantitative indicators) either into a computer-based learning environment 
or other instructional settings, it can easily be applied for individual knowledge diagnosis. The SMD Technology’ has 
been implemented as a cross-platform application which enables an easy integration into a computer-based learning 
environment. Therefore, the instructional designer may choose which components of the SMD Technology’ should be 
applied for an adequate knowledge diagnosis. The quantitative indicators could provide instant longitudinal 
information about the individual learning process. The indicators (Surface, Matching, and Deep) provide multiple 
information about changes in the knowledge structure and domain-specific knowledge acquisition. Depending on 
the results of the SMD Technology >, the learning environments will provide specific feedback or other instructional 
materials to foster future learning processes (see Ifenthaler, 2006). On the other hand, the graphical re-representation 
of the SMD Technology’ can be easily applied for individual feedback on specific tasks. The instructor could use the 
re-representation at a specific point during the learning phase to discuss the strength and weaknesses of a learner’s 
learning process. Additionally, the similarity and contrast model provide further feedback materials. 

Another use of the SMD Technology’ in the field of learning and instruction could be various fields of self 
assessment. As self assessment has the ambitious goal of making judgments about a learner’s own learning process, 
the feedback of an automated system should be very sensible to changes in the learner’s knowledge structure. As 
discussed above, the quantitative indicators and/or graphical re-representations of the SMD Technology’ could be 
applied for self assessment. A learner could receive quantitative information about his or her learning progress after 
working for a defined period with a computer-based learning environment. Additionally, the graphical re- 
representation could provide descriptive information about the learner’s knowledge structure. Furthermore, the 
similarity and contrast representation could elicit differences between previous points during the learning process or 
other learners or experts. This feature could therefore easily help to avoid the construction of misconceptions during 
self assessment phases. The major advantage of the SMD Technology for self assessment is the automated and 
instant generation of desired results. When learners receive the results of self assessment directly, their motivation to 
continue with the learning environment may be obtained longer than with other options of self assessment. 

Finally, the SMD Technology’ could be applied for analysis of knowledge management processes. 
Individuals may use the quantitative indicators and or the graphical re -representations to compare it with other team 
members while working on a project. Also, the affordances of a task could be compared with the individual 
understanding of the task and gaps could be identified to solve it effectively. Another application of the SMD 
Technology for knowledge management could be the communication of individual or group knowledge for better 
cooperation and understanding with other members or groups of a project team. Further applications could include 
knowledge identification, knowledge use, and knowledge generation (see Tergan, 2003). 



Conclusion and Further Perspectives 

The new developed SMD Technology’ is based on the theory of mental models (Seel, 1991) and graph 
theory (Flarary, 1974) and captures key latent variables associated with human learning and cognition. Graphical 
representations such as concept maps or natural language expression can be analyzed on three different levels. These 
levels help to describe individual knowledge structures from a relational, structural, and semantic point of view. 
Additionally, graphical re-representations of the SMD Technology’ provide further information regarding the 
externalized knowledge structures of a person. 

The objectivity, reliability, and validity of the computer-based and automated SMD Technology’ were 
investigated in three quasi-experimental studies. The results show a high reliability and validity in all indicators. 
Based on our findings, we developed further ideas for developing new features for the SMD Technology. These 
developments will include a tool for constructing concept maps, new techniques for describing the constructed 
models, and automated statistical reports. 

Nevertheless, the SMD Technology or parts of it (graphical re-representation; quantitative indicators) can 
be easily integrated into various applications. The tool can be used not only in mental model research, but also in 
various fields of learning and instmction. Beyond this, such computer-based and automated instmments could also 
prove to be beneficial in a wide span of other fields of research on technology and instructional development. 
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Abstract 

Critical and creating thinking concepts are defined using Bloom’s Taxonomy (1956) and the Garrison, Anderson and 
Archer (2000) “community of inquiry” model. The authors identify the relationship of cognitive presence with critical 
and creative thinking. The purpose of this paper is to identify and summarize the best practices associated with designing 
instruction that helps to build the critical thinking skills for online students. Selected literature related to building critical 
and creative thinking skills through online environments are reviewed. The authors then provide three examples of 
instruction from online courses at one south-eastern university that reflect the best practices noted for developing critical 
and creative thinking skills for both undergraduate and graduate level online courses. 

Moving teaching and learning to an online environment can pose many challenges. This is especially true when 
there is a need to develop students’ critical and creative thinking skills while in an online course. Critical thinking 
involves logical thinking and reasoning including skills such as comparison, sequencing, cause/effect, deductive and 
inductive reasoning, predicting, planning, and critiquing. In order to develop critical thinking skills in students, research 
has shown there is a need to develop instructional content and teaching styles that help to develop students’ reflective, 
creative and self-guided learning (Garrison & Anderson, 2003; Boris & Hall, 2005; Bloom, 1933) . Online courses 
currently rely heavily on text-based information to provide instructional content and communications with students. In 
addition to providing well written text-based content and using sound questioning techniques, educators need to know if 
more can be done to develop higher-order thinking skills of students through the organization and design of the online 
course. Educators would benefit from a summary of best practices that help to identify organization, questioning, 
practice and instructional techniques that will develop critical thinking skills for students in an online environment. 
Therefore, the primary focus of this paper is to identify a list of “best practices” when moving online students above the 
application level of the cognitive domain and on to the levels of learning where the student can develop skills to evaluate 
and create new knowledge. 

This article provides a basic description of critical thinking. Current literature is summarized to identify the 
“best practices” for developing critical thinking skills in an online course. Three different examples of instruction that are 
used in online courses to develop critical and creative thinking skills for online students at one university is explained. 
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Literature Review 

Benjamin Bloom (1956) and his colleagues developed a classification system that helps to depict the cognitive 
domain. This classification is known as “Bloom’s Taxonomy” 

(see figure 1). The cognitive domain involves 
knowledge and the development of 
intellectual skills. This includes the 
recall or recognition of specific facts, 
procedural patterns, and concepts that 
serve in the development of intellectual 
abilities and skills. Bloom included the following 
categories of intellectual skills in his original taxonomy: 
knowledge, comprehension, application, analysis, synthesis, and 
evaluation. His taxonomy represents the levels of knowing that a 
learner progresses through as they begin to develop higher order 
thinking skills. These categories of knowing can 
be thought of as degrees of difficulties. 

That is, the first level must be mastered 
before the next one can take place. 

However , learners do not move through the levels of 
knowing strictly in a linear fashion. Students need a foundation 
in which to build knowledge on. However, as the students 
acquire new knowledge and inevitably ask more questions, they 
need the flexibility to step back to the first level of knowledge 
building at any point in the learning process. Therefore, as 
learners develop skills or knowledge, they 
can move on to the next level to develop 

new intellectual skills or, if needed, they can repeat any previous levels. 




Figure 1 

Bloom’s Taxonomy (1956) 



Bloom's work is still very relevant to current theories. 

Anderson and Krathwohl revised Bloom's original taxonomy 
by combining both the cognitive process and knowledge 
dimensions. Anderson and Krathwohl (2001) made 
modifications to Bloom’s original taxonomy to come up with a 
revised taxonomy of the cognitive domain presented in figure 2. 

The “revised taxonomy” includes all of Bloom’s 
classifications from the 1956 model as well as a new category 
described as “creating” new knowledge. The new taxonomy now 
includes both critical thinking (evaluating) and creating new 
knowledge as the highest categories of knowing. The new 
classification of “creating” includes learning activities such as £ 

generating, planning, or producing new information or 
knowledge. This new expanded taxonomy is helpful to both 
instructional designers and educators to write and revise Revised Taxonomy (2001) 

learning objectives that including developing critical and 
creative thinking skills. 

When educators use either of these taxonomies to plan instruction they may find that they generally are 
planning instruction that covers the first two or three levels of learning: knowledge, comprehension and application. 
However, some may find that they really have not developed instruction that will move students into higher order 
learning that demonstrates analysis and above. Educators should plan to provide opportunities for students to review their 
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basic knowledge, comprehension, and application abilities and then the instmctional/practice opportunities should then 
move the students into the higher levels of knowing that include synthesis, evaluation and creating new knowledge. 

Online learning has the properties and tools needed to support higher-order learning and create the cognitive 
presence congruent with deep and meaningful learning outcomes (Garrison, 2003). The asynchronous and virtual 
nature of online learning calls on learners to be self-directed and to take responsibility for their learning. By offering 
instruction online, students will be able to participate in a more self-monitored learning environment that offers 
flexibility to the student as well as increased access to information. Therefore it is important for instructors to utilize 
the online course tools that are generally available through various course management systems such as: online 
discussion board, chats, groups, blogs, wikis, etc., to encourage students to be involved in their own learning process 
through the establishment “presence”. To help to demonstrate this, Anderson (2004,^)3) summarizes that Garrison, 
Anderson and Archer (2000) conceptual model of online learning that is referred to as the “community of inquiry” 
model: 

This model postulates that deep and meaningful 
learning results when there are sufficient levels of 
three component “presences.” The first is a sufficient 
degree of cognitive presence, such that serious 
learning can take place in an environment that 
supports the development and growth of critical 
thinking skills. Cognitive presence is grounded in and 
defined by study of a particular content; thus, it 
works within the epistemological, cultural, and social 
expression of the content in an approach that supports 
the development of critical thinking skills. The 
second, social presence, relates to the establishment 
of a supportive environment such that students feel 
the necessary degree of comfort and safety to express 
their ideas in a collaborative context. The absence of 
social presence leads to an inability to express 
disagreements, share viewpoints, explore differences, 
and accept support and confirmation from peers and 
teacher. Finally, in formal education, as opposed to 
informal learning opportunities, teaching presence is 
critical for a variety of reasons. 

Cognitive presence, social presence and teaching presence are all important preconditions in helping the learner 
to feel comfortable in the learning environment. Garrison stated that establishing cognitive presence online represents a 
significant shift in the design and delivery of an educational experience (2003). As such, the guidelines for designing 
instruction that develops effective learning online require a significant shift in the educator’s thinking. Educators must 
move away from information dissemination to one of collaboratively constructing meaning and understanding (Garrison, 
2003). The practical challenge then is to design the learning activities that provide the right balance and integration of 
reflection and collaboration. 

In order to develop students’ critical thinking skills in an online environment. Garrison reports that there are 
two very important effective practices that must be developed in an asynchronous online course: 1) Provide students the 
opportunity to reflect; and 2) Provide students the opportunity to collaborate with others, (2003). Through reflection, 
instructors must use written communication effectively, and provide learners the opportunity to revise and refine their 
comments and ideas. Reflection has to do with the state of learning and a learner’s own knowledge, experiences, and 
thought processes (Dewey, 1933). To Dewey, learning was inducing reflection through questions and actively 
monitoring this inquiry for the purpose of achieving understanding (1933). Critical thinking is generally agreed to 
include the evaluation of the worth, accuracy, or authenticity of various propositions, leading to a supportable decision or 
direction for action (Jones, 1996). 

This type of reflection is normally referred to as metacognition. Metacognition is an important concept in 
cognitive theory. “Metacognitive skills include taking conscious control of learning, planning and selecting 
strategies, monitoring the progress of learning, correcting errors, analyzing the effectiveness of learning strategies, 
and changing learning behaviors and strategies when necessary," (Ridley, D.S., Schutz, P.A., Glanz, R.S. & 
Weinstein, C.E., 1992). Metacognition consists of two basic processes occurring simultaneously: 



Figure 3 Community of Inquiry 




Teaching presence 
( structure process ) 
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1) Students monitor their progress as they learn, and 

2) Students make changes and adapt their learning strategies if they perceive they are not meeting 
their learning goals. 

Metacognition is about self-reflection, self-responsibility and initiative. Metacognition refers to knowledge of one's 
own thinking processes and products or anything related to them. It is "thinking about one's own thinking. At the 
same time, students must be responsible for goal setting and for managing their own time. 

Perhaps the most effective practice in establishing an online cognitive presence congruent with higher- 
order learning is for the teacher to model reflective inquiry (Garrison, 2003). This is best done with the teacher 
objectively providing commentary and insight into their thinking process (i.e., thinking out-loud). The purpose is to 
increase metacognitive awareness - as stated above, a precondition for critical thinking and self-direction. Modeling 
reflective inquiry provides learners with concrete examples of how to approach subject matter for purposes of 
constructing personal meaning. Students learn how to manage and monitor their own learning. They gain the ability 
and confidence to be self-directed learners. In this regard, the teacher must participate in, but not dominate, 
discussions. 

Modeling reflective inquiry and increasing metacognitive awareness can be greatly assisted by explicitly 
sharing a model of the thinking and learning process such as practical inquiry. Insight into the phases of inquiry and 
learning can help the learner appreciate whether they are in a problem definition stage, searching for relevant 
information, connecting ideas for meaning, or confirming understanding. Metacognitive awareness provided by such 
models can be an important tool in acting confidently and effectively through the selection and employment of 
appropriate strategies. This combined with teachers sharing their thinking process can be of considerable help to 
learners to develop metacognitive strategies and abilities and become reflective, self-directed learners. 

The first challenge is to establish a community of inquiry in the online environment where learners feel 
connected and are cognitively engaged; and where there is a community that supports and encourages ideas to be 
critically analyzed and meaning negotiated. The discourse, however, must be purposeful and focused. The instructor 
must be able to interject new ideas, diagnose misconceptions, and move the discussion toward resolution that may or 
may not be predictable. The role of the instructor (as the facilitator) “goes beyond a neutral weaving of participants’ 
contributions” (Garrison & Anderson, 2003). Clarifying, explaining and summarizing are legitimate functions of a 
facilitator. As long as this direct intervention is constructive, open communication is not threatened. Garrison states, 
“lecturing online or simply providing access to information is a complete misuse of asynchronous learning 
networks,” (2003). 

One important technique to use in an online environment is to allow students to moderate their discussion 
in small groups. This will actively engage most learners in a committed and free manner. The key is for students to 
report back their progress or conclusions. In this way, they receive appropriate feedback from all participants and 
confirmation of their understanding. By providing this increased responsibility and control, learners are encouraged 
to become more self-directed. The same technique can also be used for group projects, which is an excellent way to 
have learners collaboratively apply their new knowledge. 

Finally, the use of online discussions can be very productive tools when used to develop critical thinking 
skills of online students. When instructors select a critical thinking strategy to use during online class or small group 
discussions, it is important for the instructor to frame the entry of the discussion so that students are able to focus 
their postings to the discussion topic and encourage students to offer deeper dialogue that contributes to the 
discussion. Collison, Elbaum, Haavind and Tinker confirm that instructors should inform students of the standards 
and expectations of discussions before the online discussions begin. Also, identify and highlight productive lines of 
discussions when they occur. Also, instructors should provide examples of previous discussions as good models of 
online discussions whenever appropriate. Instructors should use re-direction strategies to keep the focus of the 
discussion on the topic at hand (2000). 



Best Practices Summarized 

Developing critical thinking in an online course is different from developing these same skills in a face-to-face 
course simply because of the online tools that are utilized to communicate and share instructions and feedback with 
students. When designing the online course, educators need to be familiar with the online instructional and 
communication tools that are available and utilize the tools in order to implement the best practices of developing critical 
thinking skills for students online. Educators can use many of the course management tools such as group sites, blogs, 
discussion boards, and file exchanges to help to develop higher-order thinking skills of their students. 

Here is a list of the top ten best practices as they can implemented through the use of course management tools 
that are generally available in online course: 
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Best Practice 


Course Management Tools 


Uses for Building Critical Thinking Skills Online 


1 . Communicate current 
information and resources 
to learners regularly 
throughout the course. 


Announcements page, Introductory 
Modules, Discussion Board 


Demonstrate active instructor participation, share 
expectations with students, provide navigational 
instructions, inform students of current 
information and resources; serves as a motivator to 
students by keeping information current 


2. Share expectations of 
students’ need to build 
metacognition skills in the 
first week of class 


Introductory Modules, Discussion 
Board 


Metacognition is an important concept in cognitive 
theory. State early in the course the expectations of 
students to take a conscious control of their 
learning, monitor their own progress, and selecting 
strategies to help them to learn 


3. Provide a well organized 
and resourceful online 
learning environment that 
explains content clearly 


Learning Units and/or Modules 


Organize content, resources and assignments into 
learning units or modules provides consistency in 
layout of information; Instructor’s can provide 
examples and a “reflective” model of the process 
the students will experience and help to explain 
abstract concepts as they come up in the course 


4. State learning objectives 
and expectations from the 
beginning 


Learning Units and/or Modules 


Communicate to students what they are expected 
to learn and how their learning will be evaluated 
(provide rubrics when appropriate) 


5. Provide opportunities to 
students to build and/or 
review foundation 
knowledge on their own 
when needed 


Online Assessments/Quizzes 


Provide students immediate feedback in 
knowledge/comprehension/application levels of 
intelligence; Students can self-monitor and 
moderate behaviors as needed to learn 


6. Engage students in 
processing their thoughts 
through “reflections” 


Blogs, Online Journals 


Engage students in processing thought through 
opportunities to reflect on their learning 
throughout the course; Provides the instructor a 
means for sharing feedback to individual students 
(adding comments to individual postings and 
keeping it confidential) 
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Best Practice 


Course Management Tools 


Uses for Building Critical Thinking Skills 
Online 


7. Question students regularly 
about their “thought process” 


Discussion Board; Blogs - 
Asynchronous Communication 

Or 

Live Chat Session - 
Synchronous Communication 


Used for engaging students in thought 
process; Use of questioning techniques that 
encourage critical thinking can be 
implemented; Allows time for students to 
“look up” references and information and 
respond within a given time frame; Allows 
for open communication for students to ask 
questions, respond to other views; 
Encourages active participation of student 
and instructor alike; Could serve as a 
motivator for learning 


8. Provide the means of 
building peer relationships, 
online interactions, and 
opportunities to collaborate 
with others online 


Groups Page (using the group 
discussion board and group file 
exchange tools), Wikis (team web 
sites) 


Provides means of communicating and 
working with other students online to 
complete assignments and/or projects; 
Provides the instructor a means for sharing 
feedback with group members where only the 
active group members can see the 
information 


9. Provide feedback and 
guidance to students (requires 
active instructor participation 
each week) 


Assignments and Safe Assignments; 
Group Discussion Boards, Class 
Discussion Board 


Used for submitting students’ work; 
Instructors can provide feedback and 
guidance on specific 
assignments/participation to students 


10. Model the learning process 
whenever possible 


Learning Units and/or Modules or 
Discussion Boards, Wikis, or Blogs 
(journals) devoted to modeling 
process and instructor reflections 


Instructor’s can provide examples and a 
“reflective” model of the process the students 
will experience; Use concrete examples to 
demonstrate processes and/or abstract 
concepts 


11. Provide Critical Thinking 
Rubrics 


Display within Modules folder or in 
Course Documents command button 


Allow the students to see how they will be 
evaluated in demonstrating Critical Thinking 
in the course and/or assignment 



Three Examples: Developing Critical Thinking Skills in Online Courses 

Example 1: Online Graduate Level Library Science Course 
Genre: Modern/Contemporary Fiction 

Objective: The objective for this assignment is for LIB 501/701 students to engage critical thinking based on a mini- 
author study focusing on Gary Paulsen and his books. And to find similar books based on Paulsen’s titles to 
motivate students to read other authors. 

Some verbs used for indications of critical thinking are based on Bloom’s Taxonomy for Educational Objectives. 
Some examples of these verbs include 

• Compare, contribute, deconstruct, organize 

• Double check text, critique, judge base on criteria, justify statements based on text and other evidence 

• Design, create, plan, produce 

Mini Author Study: Author studies are an interesting teaching method for engaging readers, of all ages, in making 
connections between the author’s life and the books they write. An interesting question to ask is who are authors? 
This question is a board question that allows readers, of any age, to investigate the lives of authors as real people and 
find the challenges, successes, and other interesting aspects about authors. Also, an author study allows readers to 
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seek out other books by the same author or similar type of books by other authors. Finally, investigating authors may 
motivate readers to write. 

Post to the discussion forum: Gary Paulsen and his books. 

Remember to post your initial response and then engage in conversations on three different days with at least one 
response to others. 

Example 2: Online Undergraduate Level Communication Course 
Communication Group Critical Thinking Assignment: 

Many times in this class and others, you have been asked to be problem-solvers. Often this is done as an 
assignment. It may be a case study in which certain conditions are given you that are part of a problem - perhaps in 
an office or a company. You are asked to suggest solutions to the problem that is already stated. 

In this assignment, the assignment will be somewhat reversed. It has been said by many that in the world of work, 
we need not only problem-solvers, but problem-finders or problem-recognizers. In other words, your group will 
predict a problem. The problem has not happened as yet. We know that many very successful organizations and 
companies are successful because they do not wait until something bad happens or there is a major problem. 

Instead, people in the organization identify the potential problem before it significantly inserts itself into operation 
of a company, team, process, etc. Others find something happening and identify a reason why a negative 
situation/event will occur if that is left unattended. In other words, a clue is discovered that influences the system 
negatively somewhere down the line. For example, an employee might note that if the process for manufacturing 
widgets continues as is, fewer and fewer widgets will meet tolerance standards. His fellow employees note that “if 
this continues, fewer good widgets will be made and there is going to be a big problem with output and the future of 
this company.” Therefore, a clue has been identified and a problem has been predicted to occur from that clue. 
Another real world example involves the storing of credit card information on laptop computers by credit card 
companies. “If that computer is stolen, thousands of credit card holders could suffer false charges.” Again, a major 
problem has been predicted based upon a clue or indicator. 

Y our task as a group is to identify existing circumstances that could lead to a major problem involving some aspect 
of communication. Look for events in the world news, in the state, in Richmond, on campus. Your group will list 
the “clues” or circumstances that could lead to a problem. The potential problem must have a communication aspect 
to it. state the sources for your “clues,” 

Example: At Disney World and Disneyland, it was noticed by Disney scouts that because certain rides were so 
popular, they produced a long line of people waiting (and complaining). What was the problem? People didn’t stop 
waiting nor did they stop going to the ride. Did they enjoy themselves as much as if the line was short? Probably 
not. Disney folks, however, saw this as a problem - a problem for “the happiest place on earth.” So, they provided 
several solutions. First, they “snaked” the line of people. That is, they no longer put them into a straight line where 
each person just saw the back of the person ahead of them. Instead they wound them back and forth so that each 
person’s view changed and gave the impression of moving (making progress toward the ride). They also provided 
shade and a gentle water mist at certain points in the line to relieve people from sun and heat. They also provided 
(at a cost) water bottles that sprayed a mist and directed it with a fan. Finally, they created the “fast pass” that 
allowed those who wanted to, a ticket with a stated hour when they would be able to go to “the head of the line.” 

Get your fast pass in the morning for 2 PM in the afternoon and you were able to avoid a long line for your ride. 
Notice here that the “problem” was discovered through observation. Ultimately, the solution had an effect on 
people’s satisfaction and was frequently communicated to others that were selecting a place to vacation. 

Example: Recently, Jerry Lewis held his traditional Labor Day telethon for MD. During the program he uttered a 
“gay slur” as a part of a joke. Likewise, a cast member of “Grey’s Anatomy” use an anti-gay term in reference to a 
fellow cast member during a public interview. In another case, a radio talk-show host used a derogatory term on air 
describing the members of a black women’s basketball team. These incidents upset many people. A question - does 
this communication indicate a real anti-gay or anti-black belief on the part of the person who told the “joke” or said 
the words. What is the effect of such utterances? Given these circumstances and events, is there a problem and, if 
so, what part of the problem can be contributed to communication. 
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Your task: 

• Select a group leader and consider roles to establish for the task. 

• Explore internet, news magazines, niche magazines, your observations, and/or your environment for 
potential problem situations or issues. The task is to find instances (clues) to a problem in the making. If 
it has been stated and acknowledged as a problem already, you are too late. For example, parking at EKU 
has been a stated problem to the extent that there has been a study and a parking master plan to solve the 
stated problem. Therefore, this will not qualify as a problem for this particular assignment. 

• Asa group, decide on the potential problem that your group will identify. 

• Classify the communication clues and analyze the potential problem as to what your group believes to be 
its causes. For example, a cause or clue to the impending problem may be a wrong or unreliable source of 
information, the lack of a qualified receiver , an unclear message , a wrong channel , or a lack of feedback 
(see Chapter 1 in your text to review other aspects of communication that may contribute to a problem). 

• Record (and cite the sources for) your “clues” to the potential larger, more comprehensive, problem. 

• Carefully define the “problem.” 

• Recommend three different solutions to the problem. The solutions will be the product of your group’s 
thinking. 

Prepare your analysis either in a paper or in a Wiki by using the Blackboard Wiki Tool. 

Example 3: Orientation Course - First Year Students 
Case-studies and branching scenarios 

Case-based e-learning challenges learners to gather information from various sources in order to make meaningful 
decisions. Information can be structured in any format such as text, graphics, hyperlinks, audio, or video. The case 
format encourages students to look at an issue from multiple perspectives and helps to develop their decision making 
abilities. Branching scenarios can be included in the case to enable learners to see the consequences of their 
decisions, making the experience more memorable and effective. 

Instructional Product: Students, serving as peer advisors, will analyze information presented in a specific case, 
access appropriate information and/or resources and engage in problem solving techniques. 

Overview of Instructional Process: 

• In this sample, the learner plays the role of a peer advisor. To make a correct analysis and recommend 
appropriate actions to their classmate, the learner must review relevant information about the student’s 
history and current academic situation. Next the peer advisor must collect information or access resources 
that will help the “advisee” while using problem solving techniques. 

• The information resources are presented as simple text, but could easily be augmented to incorporate 
graphics, audio, video, or external resources, such as patient charts. 

• Learners will be “reflective” in their decision making process and demonstrate metacognitive skills as they 
progress through the case and problem solving activities. 

Conclusion 

Educators can promote critical thinking skills by developing course materials and activities that reinforce 
metacognitive skills. Educators who want to develop student critical thinking skills will need to develop a learning 
environment (online) that encourages students to ask questions, engage in reflective thinking and self-directed 
learning. At the same time, if is very important for the educator to model the reflective skills needed. Finally, the 
online discussions are important tools used to develop critical thinking skills for online students. Therefore, 
instructors should continue to seek out effective strategies and online tools that are used in developing rich and 
thought provoking discussions and reflections with their students online. Distance educators should work with 
instructional designers to develop online content and instructions that foster critical thinking skills. 
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Cognitive presence, a sense of “being there” cognitively, has recently been considered an important factor for 
students’ engagement in e-learning. There is, however, no widely accepted quantitative measurement scale for 
cognitive presence because most studies on cognitive presence have been conducted qualitatively. Therefore, 
existing theories on cognitive sector-related research regarding cognitive presence were reviewed and a new 
measurement scale for cognitive presence was developed. This study tested reliability and validity of the 
measurement scale of cognitive presence with 305 undergraduate students. Three major factors of cognitive 
presence were perceived levels of: 1) understanding content, 2) constructing knowledge, and 3) managing learning 
resources. 



Introduction 

Cognitive presence, a sense of “being there” cognitively, has been mainly researched by constructivists who 
stated that cognitive presence reflects higher-order knowledge acquisition and application in online learning 
(Garrison, 2004). Cognitive presence is also an important factor in facilitating learners’ engagement and in affecting 
a learner’s level of achievement and satisfaction (Wang & Kang, 2005; Kang, 2005). 

Despite the importance of cognitive presence for successful online learning, there is no widely accepted 
measurement scale. Therefore, a new scale for measuring cognitive presence was developed based on the review of 
existing theories on the cognitive factors activating learners’ engagement in an online learning environment. Then, 
reliability and validity of the new scale were tested with 4 1 8 undergraduate students in an e-learning environment. 

In this study, the developed scale was retested with 305 undergraduate students in online environments. 

Theoretical Background 

Studies related to online learning have begun to shift their focus to better understand the unique needs associated 
with online learners. The shift is increasing the importance of cognitive engagement in online learning (Richardson 
& Newby, 2004). One of the crucial factors for engagement in online learning is considered to be the level of 
perceived cognitive presence (Wang & Kang, 2005). In other words, engaged learners might have a higher level of 
perception of cognitive presence in an e-learning context. 

According to cognitive engagement theory (Ryan & Patrick, 2001; Bangert-Drowns & Pyke, 2002; Wang & 
Kang, 2005), there are three characteristics that engaged learners might share: understanding, constructing, and self- 
regulating in knowledge construction. First, a high level of cognitive presence should be able to facilitate 
information acquisition, information transportation, and constructing knowledge (Corno & Mandinach, 1983; Wang 
& Kang, 2005). According to the cognitive engagement theory, knowledge construction has three factors: 
information acquisition, information transformation and constructing knowledge (Corno & Mandinach, 1983; Wang 
& Kang, 2005). Based on this theory, a high level of learners’ perceived cognitive presence could facilitate 
knowledge construction (Wang & Kang, 2005). 

Second, learners with a high level of perceived cognitive presence understand learning contents well. According 
to the cognitive engagement camp, cognitive engagement is the mobilization of cognitive strategies for interpretive 
transaction (Bangert-Drowns & Pyke, 2001). In other words, engaged learners are thought to be more intellectually 
concerned with their learning tasks intensively and extensively (Bangert-Drowns & Pyke, 2002). Accordingly, 



183 




students’ engagement entails an intrinsically motivated involvement of the integrated cognitive process (Kearsley & 
Schneiderman, 1998). 

Third, learners who perceive a high level of cognitive presence manage learning resources freely. Since cognitive 
engagement is considered a core variable in a well-developed self-regulating learning process, engaged learners will 
be able to manage well resources, environment and performance (Mckeachie et al., 1986; Pintrich & De Groot, 
1990). Resource management involves the process of developing well-defined goals and scheduling the course to 
achieve the best results. Environment management is the development of a physical setting that is helpful to 
learners. Performance management includes self-effort, self-reinforcement and persistence (Mckeachie et al., 1986). 

Learners’ cognitive presence activating engagement in an online environment has been investigated from both 
constructivist and cognitive perspectives. Constructivists have defined cognitive presence as the extent to which 
learners are able to construct and confirm meaning through sustained reflection and discourse in a critical 
community of inquiry (Garrison, Anderson, & Archer, 2000). They argued that knowledge is constructed by the 
circular process of triggering events, exploring, integrating, and resolving tasks during online learning (Garrison, 
2004). Cognitive presence has also been empirically validated with respect to cognitive engagement within online 
groups as providing a theoretical background for measuring the cognitive engagement of online learning (Oriogun, 
Ravenscroft, & Cook, 2005). 

The above discussion led to the extraction of three factors of cognitive presence: understanding content, 
constructing knowledge, and learning resources management. Accordingly, the operational definition of cognitive 
presence in this study was defined as the ‘perceived level of general understanding, knowledge construction, and 
learning resources management during e-Learning.’ 



Methods 

As a result of primary factor extraction of 18 items from a survey of 41 8 students, three dimensions of 
cognitive presence were verified. To improve reliability, this study was retested with 305 undergraduate students. 
They enrolled in an online course titled ‘Design of College Life’ in the fall semester of 2006 at a large university in 
South Korea. The class lasted for eight weeks and the cognitive presence scale was distributed to participants as an 
online survey during the fourth week. 

The three main components of cognitive presence are the level of understanding content, constructing 
knowledge, and managing learning resources. Each component has sub-components with 9 items based on a 
theoretical framework. 

First, the level of understanding the contents has three sub-components: 1) consistency between content and 
objective, 2) organization of content, 3) articulation of content (Broadbent, 1958; Gagne, Yekovich & Yekovich, 
1993). For example, the following items were used; “I could organize what 1 learned with diagrams and graphs,” “I 
could explain what I learned in class,” and so on. 

Second, the level of constructing knowledge has the following sub-components: 1) information acquisition, 2) 
information transformation, and 3) constructing knowledge (Corno & Mandinach, 1983; Wang & Kang, 2005). For 
example, the following items were used; “I could collect information related to the class,” “I could make a 
connection between new information and what I already knew.” 

Third, the level of managing the learning resources has sub-components: 1) time management, 2) performance 
management, and 3) environment management (Ryan & Patrick, 2001; Zimmerman, 1990; McKeachie et al., 1986). 
For example, the following items were used; “I could do assignments by making a plan,” “I could avoid any 
distractions while studying.” 

As presented above, 27 items with a five-point Likert scale were developed via an online survey. The items 
were analyzed by an expert for content validity and modified based on recommendations. The online survey was 
conducted and 305 responses were collected. Exploratory factor analysis (EFA) was performed to verify the 
emergence of the three dimensions of cognitive presence. Principal axis factoring method was used to extract 
factors. To rotate factors, direct oblimin rotation method was used. 

Results 

The results of EFA with 27 items yielded three factors: understanding content, constructing knowledge, and 
learning resources management. The reliability of these factors with Cronbach’s coefficient alpha yielded .844, 

.809, .640, respectively. Six items showing loading lower than .40 were removed (Thurstone, 1947). As a result, 21 
remaining items are reported in Table 1. 
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Table 1: Emerged factors of cognitive presence 



Main 

components 


Remaining Items 


Factor 

loading 


Cronbach’s 

alpha 


Level of 
Understanding 
the Contents 


The content of class is that I usually want to learn. 


-.402 


.844 


The content of class is what 1 expected. 


-.404 


I think I can understand content of class good enough to 
draw diagrams and graphs. 


-.541 


I can reorganize what 1 learned my class. 


-.734 


I think I can outline what I learned my class. 


-.655 


I think I can explain what I learned my class. 


-.612 


I think 1 can discuss what I learned my class. 


-.667 


I think I can briefly summarize learning material. 


-.723 


Level of 
Constructing 
the Knowledge 


1 usually search for extra materials related to the class. 


.450 


.809 


I feel I can select materials what I need. 


.457 


I feel I can collect the information related to the class. 


.606 


I feel I can use what I learned in the class to do 
assignment. 


.645 


I feel I deeply understand what I learned. 


.507 


I feel that I’m learning in this class. 


.584 


I’m getting new perspective through this class. 


.495 


I feel I can apply what I learned in reality. 


.471 


Level of 
Managing 
the Learning 
Resources 


I feel that I can do assignments as planed. 


.468 


.640 


I know how to be helped. 


.454 


I reorganize the material for the assignment, the course 
activity, and the discussion. 


.463 


I look for the comfortable environment that I could focus 
on my study. 


.595 


I feel that I could eliminate the obstacles that disturb my 
study. 


.494 



Discussion and Implication 

We performed a second validation process on this measurement of cognitive presence based on theory. 
According to studies on the cognitive engagement camp, cognitive presence is composed of three dimensions: level 
of understanding the content, constructing knowledge, and managing learning resources. In this study, the cognitive 
presence scale was grounded in theory and validated three constructs through statistical analysis as in the 
preliminary study. 

Consequently, this study shows that cognitive presence activating learner’s engagement consists of three 
dimensions and these could be a basis for measuring the level of cognitive presence. At present, a follow-up study 
is in progress to conduct a CFA (confirmatory factor analysis). 
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Abstract 

Over the past two decades, the development of information and communication technologies (ICT) has 
moved educational systems from teacher-centered pedagogy to learner-centered pedagogy. The recent convergence 
of ICT has fostered another reform in education. Therefore, this article will examine the impacts of the ICT 
convergence on the current pedagogy, review the main principles of two current learner-centered instructional 
theories, and suggest potential modifications of these theories for the new pedagogy. 

Introduction 

Over the past two decades, the development of information and communication technologies (ICT) has 
resulted in significant changes in the global economy, referred to as globalization, the information society, and the 
knowledge economy (Kozma, 2005; Sachs, 2005; Tinio, 2005). This new technological global economy system has 
enormous implications for the nature and purpose of the entire educational system as well as other subsystems in the 
society (Abbott, 2001; Bates, 2001; Carnoy, 2004; Coupal, 2004; Dede, 2000; Lim & Hang, 2003; Lim, 2003; 

Mooij, 2004; Moore, Burton, & Myers, 2004; Molz, Eckhardt, & Schnotz, 2002; Tinio, 2005). Essentially, these 
educational changes have required a paradigm shift in instructional theories (Reigeluth, 1999). Thus, the focus of 
educational researchers and practitioners has moved from passive teacher-centered instruction to active learner- 
centered instruction (Bonk & Cunningham, 1998; Duffy & Kirkley, 2004; Reigeluth, 1999). 

In the midst of these changes, a convergence within ICT has emerged. The convergence of cutting-edge 
technologies, combining different media into one operating platform (e.g., multimedia cell phones including digital 
cameras, MP3 players, and DMB systems), has resulted in another natural reform of instruction and learning in 
order to improve learner-centered pedagogy (Erstad, 2005; Kozma, 2005). As a key notion of this reform, 
knowledge-building communities through virtual learning environments have been proposed (Barab, Kling, & Gray, 
2004; Bonk, Wisher, & Nigrelli, 2004; Hewitt, 2004; Kozma, 2005; Scardamalia & Bereiter, 1994). As part of these 
trends, this paper will examine impacts of the ICT convergence on learner-centered pedagogy and present 
suggestions for modifying current instructional theories to knowledge-building communities. 

Another New Challenge to Instructional Theories: The Convergence of ICT 

Traditional information and communications media were once separate, and their services (e.g., 
broadcasting, voice telephony, and online computer services) were operated on distinctly different platforms (e.g., 
TV and radio sets, telephones, and computers; NetTel, 2005). However, the current combination of ICT merges 
separate media into one operating platform and provides the different services as multi-services, including telecom, 
data processing, images and audio technologies. Recent examples of new convergent services are web TVs (TVs 
with internet services), multimedia mobile phones (cell phones with e-mail and WWW access, MP3 players, and 
digital broadcasting capabilities), and web casting (radio and TV programming on the Internet). This new ICT 
convergence is not limited to technology issues but has also resulted in the convergence of other social phenomenon 
such as functional, technological, economic, political, and geographic convergence (NetTel, 2005). 

One of the most significant changes brought by this ICT convergence is to shift users from content/service 
consumers to content/service creators. New development and convergences of ICT have made personal technology 
tools smaller, lighter, cheaper, and more mobile and allowed their users (e.g., students in education) to be freer, 
opener, more collaborative and creative. Learners are no longer just passive knowledge consumers who adopt 
identical services provided by systems. Now, they can create, share, and use their own new knowledge as experts do, 
whenever and wherever, through the convergence of technologies (Barab, Kling, & Gray, 2004; Kozma, 2005). This 
can be identified as knowledge creation through the convergence of ICT. 

Essentially, this new phenomenon requests another advanced pedagogical reform distinguished from 
learner-centered education. Knowledge building communities have been suggested as an ideal direction to this 
change (Barab, Kling, & Gray, 2004; Bonk, Wisher, & Nigrelli, 2004; Hewitt, 2004; Kozma, 2005; Scardamalia & 
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Bereiter, 1994). Knowledge building communities can be described in terms of key components of ICT and 
pedagogy, as shown in Table 1 . However, it should be noted that this new notion of pedagogy still includes many 
important learner-centered pedagogical features because this notion has been developed based on the current 
learner-centered paradigm, while learner-centered pedagogy has been evolved in contrast to teacher-centered 
pedagogy. 

Table 1. Dimensions of teacher, learner, and community-centered educations related to ICT, adopted or modified 
from Bransford, Brown, & Cocking (2000), Erstad (2005), Hewitt (2004), Kozma (2005), Law, Kankaanranta, & 
Chow (2005), Scardamalia & Bereiter (1994), Srivastava & Venugopal (2005), and Voogt & Pelgrum (2005) 





Teacher-centered 

education 


Learner-centered 

education 


Knowledge building 
community-centered 
education 


Level of knowledge 


Memorizing and 
understanding concepts, 
principles, and 
procedures outside of the 
context of the real world. 


Deep understanding of 
complex concepts and 
transferring learning to 
the real world. 


Creating new knowledge 
in the real world. 


Way of 

communication 


One to many 
(e.g., a teacher to 
students). 


Many to many 
(e.g., students to 
students). 


Massive to massive 
(e.g., community 
members to community 
members). 


Nature of 
technology 


Separate media 
(e.g., slides, radio). 


Separate media and 
multimedia: the 
traditional convergence of 
ICT (i.e., computers and 
the Internet). 


The new convergence of 
ICT: 

mobile and wireless 
access to the Internet 
(e.g., multimedia cell 
phones, PDAs). 


Use of technology 


No/limited use of 
traditional media. 


Limited/ flexible use of 
traditional media and 
multimedia. 


Flexible use of the ICT 
convergence. 


Leadership 


No/limited strategies on 
media. 


Limited strategies on 
media. 


Breakthrough visions of 
development creating a 
new agenda for 
education. 


Goals 


Prescribed goals. 


Overall goals. 


Locally adapted. 


Tasks 


Fixed tasks in the scopes 
and sequences. 


Complex tasks. 


Open tasks set by 
students. 


Methods 


One-way transmission of 
information. 


Project orientation and 
collaboration. 

Different approach 
adjusted to student needs. 


Dependent on activities 
and goals, not predefined. 
Collaborative process of 
building on current 
knowledge to create, new 
knowledge 


Assessment 


Teacher-directed 
Summative assessment. 


T eacher/ student-directed 
Formative assessment. 


Student-directed (e.g., 
self-assessment, 
peer assessment) 
Diagnostic assessment. 


Student roles 


Passive reproducer of 
knowledge, individually. 


Producer of knowledge, 
individually and 
collaboratively. 


Active creator of 
knowledge, 

collaboratively with good 
meta-cognitive skills. 


Teacher roles 


Active provider and 
transmitter of 
information. 


Different roles related to 
activities (e.g., facilitator, 
expert). 


Organizer of 
environments, knowledge 
challenger and learner. 


Tools/Resources 


Subject-oriented books 


Combination of books 


A wide variety of 
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supporting skills; reading, 
writing, and calculating. 


and different digital 
resources (e.g., hypertext, 
multimedia). 


resources 

(e.g., games, simulations 
resources). 


Learning 

environment 


Traditional classroom 
during scheduled class 
time. 


Traditional/beyond 
classroom during the 
course 

(e.g., off/online 
classroom). 


New ways of organizing 
learning: 

breaking with school as 
the single organizing 
principle, 

virtual platforms for 
collaboration, 
no boundary between in 
and out of classroom, 
life-long education. 


Knowledge building 


Reproduction. 


Production and inquiry- 
based. 


Based on students ideas, 
knowledge creation and 
inquiry. 


Instructional 

theories 


Direct instruction. 
Game’s nine events. 


Merill’s first principle, 
problem-based learning, 
anchored instruction. 


Knowledge-building 

community. 



The convergence of ICT has accelerated the pedagogical move forward to knowledge building 
communities. Nevertheless, it has also provided all three types of pedagogies with three advantages as follows 
(Kozma, 2005; Voogt & Pelgrum; 2005; Tinio, 2005). First, regarding the quantity of learning, ICT expands access 
to education (i.e., access to remote learning resources anytime and anywhere). Second, in terms of the quality of 
learning, ICT promotes student understanding (i.e., active, integrative, and facilitating learning). Last, regarding the 
innovation of learning, ICT fosters knowledge creation (i.e., creating, sharing, and using knowledge). These positive 
effects of ICT on teaching and learning processes may be controversial. However, there is an agreement that, at least 
in principle, ICT helps learning become more effective, efficient, and appealing (Merrienboer & Brand-Gruwel, 
2005) 

As mentioned before, learners are not passive knowledge consumers or active knowledge transferors 
anymore, but now they are innovative knowledge creators equipped with high technology tools. The focus of 
learning has changed from knowledge acquisition to knowledge creation (Kozma, 2005). To face this new learning 
arena brought on by the ICT convergence, current instructional theories should be reconsidered. Thus, in the 
following section, I will review major principles of two instructional theories grounded on learner-centered 
pedagogy, problem-based learning (PBL) and anchored instruction (AI), to propose possible modifications. 

Principles of Instructional Theories: PBL and AI 

Two instructional theories, PBL and AI, have been selected because their goals, values, situations, and 
component methods are clearly identified (Reigeluth, 1999), and their importance and applications are still 
considerable in the new virtual and mobile learning environments (Steinkuehler, 2006 and in press; Steinkuehler, 
Derry, Hmelo-Silver, & Delmarcelle, 2002). The main goals of these instructional theories are to promote cognitive 
domain (e.g., self-directed learning, collaboration skills) in face-to-face learning in K-16 settings. 

Problem-based Learning 

Problem-based learning (PBL), originally developed in medical schools at Southern Illinois University in 
the early 1970s, is an instructional theory in which students learn through facilitated problem solving using the 
processes of investigation, explanation, and resolution (Barrows, 2000; Hmelo-Silver, 2004; Savery & Duffy, 1996). 
The main principles of PBL are presented in Table 2. 

Table 2. Main principles of problem-based learning, adopted or modified from Barrows (1994 and 2000), Barrows 
& Kelson (1995), Herrington & Oliver (2000), Hmelo-Silver (2004), Koschmann, Myers, Feltovich, & Barrows 
(1994), and Saveiy & Duffy (1996) 
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Main principles 


Goals 


• Develop effective problem-solving skills 

• Construct extensive, flexible knowledge bases 

• Improve intrinsic motivation 

• Develop self-directed learning skills 

• Improve effective collaboration skills 


Problems (Tasks) 


• Be authentic, complex, ill-structured, and open-ended 

• Raise concepts and principles relevant to contents 

• Focus on learning information and reasoning strategies 

• Promote conjecture and argumentation 

• Help students engage in the learning process with their initial 
understanding 

• Require multidisciplinary solutions 

• Foster communication skills 

• Help students build extensive and flexible knowledge 


Methods/Processes 
(see Figure 1) 


Facilitator 

• Present a problem scenario 
Students 

• Analyze the problem by identifying the relevant facts from the scenario 

• Generate hypotheses about possible solutions 

• Identify knowledge deficiencies relative to the problem during self-directed 
learning 

• Apply new knowledge 

• Evaluate the hypotheses in light of what students have learned 

• Reflect on the abstract knowledge gained 


Assessments 


• Self-evaluation 


Student roles 


• Learn what they should know to solve a problem 

• Bring new knowledge to group for application to problem 

• Work in small groups 

• Negotiate ideas 

• Reflect critically the relationship between their learning and problem- 
solving goals 


Facilitator roles 
(Teacher roles) 


• Facilitate learning process and model reasoning 

• Help students leam the cognitive skills needed for problem solving and 
collaboration 

• Be an expert in good strategies, not in the content 

• Scaffold student learning through modeling and coaching 

• Monitor students involvement 

• Encourage students to externalize their own thinking and comment on 
peers’ thinking 

• Encourage students to test ideas against alternative views 

• Support students to develop ownerships 

• Guide higher-order thinking by justifying students thinking 

• Externalize self-reflection by directing questions to students 


Tools/Resources 


• Structured whiteboard 

• Student-identified learning resources 


Learning environment 


• Support and challenge students’ thinking 

• Face-to-face classroom 
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Figure 1. The problem-based learning cycle (Hmelo-Silver, 2004, p. 237) 




Anchored Instruction 

Like another family of constructivist approaches including PBL and project-based science (Hmelo-Silver, 
2004), anchored instruction (AI) was initially designed for K-12 students by Bransford and his colleagues in the 
Cognition and Technology Group at Vanderbilt (CTGV) in the late 1980s. AI refers to instruction in which learning 
materials (i.e., videodisc-based narrations) are presented in the context of the authentic events that serve to anchor 
learning (Barab, Hay, & Duffy, 1998). The Young Sherlock Project (language arts and social studies) and the Jasper 
Series (mathematics) are representative example projects of AI (CTGV, 1990, 1992, and 1997). The fundamental 
characteristics of AI are as listed in Table 3. 

Table 3. Main principles of anchored instruction, adopted or modified from Barab, Hay, & Duffy (1998), Bransford, 
Sherwood, Hasselbring, Kinzer, & Williams (1990), CTGV (1990; 1991; 1992; 1997), Hmelo-Silver (2004), Moore 
et al. (1994), and Kang (1994). 





Main principles 


Goals 


• Overcome inert knowledge 

• Foster meta-cognitive skills 

• Promote cooperative learning 

• Build multiple perspectives 


Stories (Tasks) 


• Provide complex problems in videodisc-based narrations 

• Link across the curriculum 

• Present interesting, realistic, and macro context to encourage active 
knowledge construction 

• Generate many sub-goals (at least 15) 

• Embed all fact data necessary to solve the problem 

• Present major problems to be solved at the end video 

• Open-ended 


Methods/Processes 


Teachers 

• Present the video-based stories to support comprehension 
Students 

• Identify major goals of the given story 

• Generate sub-problems that represent obstacles to the goal 

• Devise strategies to deal with various sub-problems 
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• Find all relevant information presented throughout the story 

• Generate what students need to know 

• Retrieve the information from memory 

• Scan back on the video clip to see if information is accurate 


Assessment 


• Not specific mentioned 


Student roles 


• Find top-down strategies to generate sub-goals necessary to reach the final 
goal 

• Learn the lower level skills through 

• Reflect on their own perspectives 

• Negotiation of ideas and strategies within small groups and whole class 

• Examine problems from multiple perspectives 


Teacher roles 


• Engage students’ prior knowledge 

• Model problem-solving strategies 

• Provide content instruction when needed by students 

• Guide planning and sub-goals generation 


Tools/Resources 


• Videodiscs, VCRs, and video controllers 

• Problem-specific tools (e.g., maps, pencils) 


Learning environment 


• Face-to-face classroom 



A Direction of Instructional Theory Modifications: Knowledge Building Communities 

This section will present suggestions for PBL and AI modifications in new learning environments opened 
by the ICT convergence regarding the difference between knowledge levels aimed in two pedagogies (deep 
understanding vs. knowledge creation). Also, in order to increase effectiveness, efficiency, and appeal of instruction, 
these modifications will focus on the new relationships among learners, teachers, tasks, tools, and resources in the 
learning environments, as well as instructional framework changes (e.g., type, control, focus, grouping, interactions, 
and support of learning; Reigeluth, 1999). 

First of all, in terms of instructional elements, the structure of traditional PBL and AI in the classroom is 
represented in Figure 2. 

Figure 2. The original structure of PBL and AI in the classroom 




The new instructional structure of PBL and AI in the virtual and mobile learning environment for 
knowledge building communities is shown in Figure 3. 

Figure 3. The new structure of PBL and AI in the virtual and mobile learning environment for knowledge building 
communities 
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As shown in Figures 2 and 3, there are meaningful differences among the relationships of elements in the 
two learning environments. Due to the nature of the virtual learning environment with internet access and data 
searching (e.g., Google, Yahoo), it is ambiguous to separate tasks, tools, and resources from the environment (i.e., 
virtual vs. real world) 1 . Moreover, because of the smart mobile tools with wireless Internet services, digital cameras, 
and broadcasting systems (e.g., notebook computers, multimedia cell phones, PDAs), lining the boundary of a 
classroom or school is also worthless (i.e., in class vs. out of class). 

For these reasons, whoever has these new combined tools can enter the learning environments any time and 
any place and learn or teach beyond traditional instruction. Thus, this new learning environment, open to the public, 
makes the distinction between information and instruction (non-formal vs. formal education) meaningless. In this 
sense, the notions of new learning systems created by the ICT convergence accord with the features of knowledge 
building communities. Meanwhile, the teachers’ role is significantly reduced because teachers, who once delivered 
knowledge or facilitated students’ learning processes in classrooms, now share their instruction or facilitation with 
new tasks, tools, or resources in the virtual and mobile learning environment 2 . 

Problem-Based Learning for Virtual Knowledge-Building Communities 

There are several possible modifications for applying PBL to the new learning environment framework, 
virtual knowledge building communities. First of all, learners’ ownership and empowerment should be more 
emphasized in the new PBL, as the members of knowledge building communities who share goals, values, and 
beliefs, learners should be able to create their norms, rules, or other orders (Bonk et al., 2004; Hewitt, 2004). 
Therefore, instead of instructional designers or facilitators, learners should participate in the process of designing 
PBL ’. For example, learners can (a) select problems in their own real world, not given scenarios, (b) devise small 
teams in the community following their decisions, (c) allot generated hypotheses to each team considering their 
interest and expertise, (d) evaluate processes and products of solutions on their negotiated rules, and (e) reflect and 
provide feedback on the values of created knowledge according to their judgments. Through these active 
involvements in the decision-making process of communities, learners can have strong senses of identity and engage 
more in PBL learning activities (Carabajal, LaPointe, & Gunawardena, 2003). 

Meanwhile, this empowerment of learners implies that the role of facilitators or instructional designers 
diminishes in significant ways. The focus of control in the new PBL does not belong to them anymore. Thus, their 



1 Dotted lines imply open-ended boundaries. 

2 No shadow means reduced teacher’s role. 

’ This role change can be explained as the change of Division of Labor in the activity system of a knowledge 
building community (Hewitt, 2004, p. 217). 
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new main roles should be changed to environment organizers, knowledge challengers, or expert learners who 
contribute to knowledge creation as community members (Erstad, 2005). 

On the other hand, discourse focusing on problems should be at the center of the new PBL in virtual 
learning environments. Community knowledge data are created by learners who participate in discourse activities 
(Scardamalia & Bereiter, 1994). In this sense, the ICT support should be designed to enhance reasoning and provoke 
critical thinking (Merrienboer & Brand-Gruwel, 2005; Merrienboer & Martens, 2002). Thus, it is essential to 
provide specially designed discourse environments (e.g., asynchronous bulletin boards, blogs) that can help learners 
construct their inquiry process in light of a problem rather than a topic (Scardamalia, 2003; Scardamalia & Bereiter, 
1994). As the other benefits of this new ICT-based discourse, (a) eliminating turn-taking problems, (b) encouraging 
peer commentary and notification, and (c) opening entry points for all ages and ability levels can be obtained 
(Scardamalia & Bereiter, 1994). 

Also, learners and facilitators should know how to think critically in discourse, which has been emphasized 
as one of meta-cognition skills for knowledge creation (Garrison, Anderson, & Archer, 2000 and 2001). For 
example, key critical thinking processes can be explained as guidelines before generating problems in the PBL 
(Garrison & Cleveland-Innes, 2005). These guidelines can help discourse participants realize how to identify the 
problem (e.g., trigger phase), how to expend their initial ideas through brainstorming (e.g., exploration phase), how 
to construct meanings from generated ideas (e.g., integration phase), and finally how to implement proposed 
solutions (e.g., resolution phase; Garrison, Anderson, & Archer, 2003). 

In addition to cognitive presence of discourse, special strategies to enhance social presence of discourse 
should be designed in the new PBL because successful social networking among learners is a core of community 
evolution, especially in virtual environments (Carabajal, LaPointe, & Gunawardena, 2003; Riel & Polin, 2004). This 
notion should be increasingly considered in the case of broader communities that consist of massive members 
beyond classroom, school, or nation (Scardamalia & Bereiter, 1994). 

Anchored Instruction for Knowledge Creation in Virtual and Mobile Learning Environments 

Basically, the concepts about knowledge building communities aimed in new AI are the same to those in 
modified PBL. However, unlike modified PBL, the intervention of instructional designers or teachers does not 
decrease significantly in terms of prescribing stories to anchor the instruction (i.e., embedding fact data in the stories 
during the design process). For this reason, the scope and boundary of the new AI may not expand beyond the 
formal education systems, unlike the new PBL. Therefore, modification points for AI (anchored instruction) will be 
indicated only regarding technological aspects of AI (e.g., AI using smart virtual and mobile ICT equipments) 
instead of global community aspects. 

First, up-to-date ICT should be used to improve instructional effectiveness and efficiency in AI (Dede, 
2000; Merrienboer & Brand-Gruwel, 2005). For instance, cutting-edge ICT display tools can replace videodisc- 
based narrations as learning materials to present macro-contexts in AI. Stories and narratives are very appealing 
instructional methods because learners can easily associate with the contexts, plots, and elements manifested in them 
(Bruner, 1996; Hung, Tan, Cheung, & Hu, 2004). Thus, as one of the key instructional characteristics in AI, stories 
have values even in the new learning environments. However, in terms of the effectiveness and efficiency of 
instruction (Reigeluth, 1999), the story delivery should be updated by adopting new ICT tools, such as digital video 
clips through the Internet (e.g., YouTube.com) or MDB services through multimedia cell phones. With these new 
methods of delivery, learners can return to the stories anytime and anywhere and are not confined to a certain time 
and space by VCRs or TVs. This mobility can increase the possibility for learners to catch the final goal and sub- 
goals from the stories. 

Also, in order to generate new knowledge, more creative activities for learners should be designed in the 
AI. Through the meaningful activities, learners can discover what they need to do to reach the final goal and 
accumulate what they find in the activity process as creative knowledge (Jong & Joolingen, 1998). For example, 
instead of embedding all fact data in the stories, only clues relevant to the facts can be presented. The clues should 
indicate clearly how and where learners can find all the fact data necessary to achieve the goal. To find these fact 
data, learners can go to the fields in their real worlds (e.g., vegetables in the grocery store, flowers in the garden) 
with appropriate mobile ICT tools (e.g., multimedia cell phones, smart PDAs with Internet access, and digital 
cameras) or search for the data in the Internet (e.g., Google, Yahoo). After finding the necessary data, learners can 
immediately upload, share, and use them in various data forms (e.g., digital photo images, movies, texts) through the 
mobile ICT equipments. These real, live, and confirmed data from the fields or the Internet are new knowledge 
created by learners. In addition, during the process searching for fact data in the real world, learners can foster their 
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self-monitoring and meta-cognitive skills (Kang, 1994). Moreover, they can link authentic multi-knowledge across 
curriculum beyond given problem situations (CTGV, 1990 and 1992). 

On the other hand, as addressed above, teachers or instructional designers should devise anchored stories 
with appropriate clues, considering available learning environments and tools. Also, teachers can be involved in the 
knowledge building processes with learners. Nevertheless, their roles are relatively reduced compared to the original 
AI. 



Conclusion 

This paper has examined the change of the current pedagogy informed by the new ICT convergence, 
reviewed the main principles of two learner-centered instructional theories (PBL and AI), and furthermore suggested 
the potential directions of modifications in these theories toward knowledge building communities. 

Learners empowered by cutting-edge technologies refuse remaining as passive knowledge consumers. 

Now, they are moving forward to knowledge creators (Kozma, 2005). The convergence of ICT makes the boundary 
of physically separated learning environments, tools, and resources meaningless in instruction. Also, conceptual 
distinctions between virtual and real worlds, in and out-of classroom, and formal and non-formal education become 
vague. Nevertheless, the features of new learning environments brought by the ICT combination will satisfy the 
necessary conditions for the development of knowledge building communities. 

Therefore, to face this new learning arena with great potential, current instructional theories should be 
improved in terms of effectiveness, efficiency, and appeal (Reigeluth, 1999). However, their modifications should 
be confirmed through designed-based research approaches in the fields (Barab, 2004; Reeves, 2000; Reigeluth & 
Frick, 1999; Richey, Klein, & Nelson, 2004; Wilson, 2005; Winn 2002). Also, the relationships between 
instructional methods and situations should be investigated to present alternative methods (Reigeluth, 1999). Finally, 
the identity and role of instructional designers as changed by this new learning pedagogy should be reseated and 
clarified. 

Groups are social systems (Carabajal, LaPointe, & Gunawardena, 2003). From this perspective, the 
relationships of the various components in education should be carefully considered to reform the whole educational 
system and furthermore change the super-system, our society (Banathy, 1992; Dede, 2000). 
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A Comparison among Facilitation Strategies to Promote Critical 
Thinking of College Students in Online Discussion 

Nari Kim 
Indiana University 

Introduction 

Why should we pay attention to critical thinking of college students? Developing critical thinking 
skills is essential in all K-16 educational settings. However, critical thinking has been emphasized in higher 
education more often because it is a prime aim of American postsecondary education targeted for future mid- 
career knowledge workers (Archer, Garrison, Anderson, & Rourk, 2001; Hagedorn, Pascarella, Edison, 

Nora, & Terenzini, Hegedorn, 1999). 

Since the middle of the 1980s, many national reports such as Involvement in Learning (1984), 
Integrity in the College Curriculum (1985), National Education Goals Panel (1991), and National Goals for 
Education - 2000 (1994) have explicitly underscored critical thinking as an essential outcome of 
undergraduate education (Facione, Facione, & Giancarlo, 2000; Hagedorn et ah, 1999). The key argument of 
these reports is that graduates should be equipped to deal with the demands of a rapidly changing work 
environment in the real word. However, this should be obtained through the acquisition of core skills to 
transfer across a wide rage of situations (i.e., cooperative, communication, and problem-solving skills 
merged into critical thinking skills) rather than through the memorization of factual information and specific 
content materials to become forgotten or outdated (Bennett, Dunne, & Carre, 1999; Tsui, 2000). In fact, the 
need for critical thinking skills in the work place has been reported through numerous surveys and interviews 
of employers who have a desire for graduates with these generic core competencies as personal transferable 
skills (Bennett, Dunne, & Carre, 1999; Harvey, Moon, & Geall, 1997; Slee 1989). Recently concurring with 
the pedagogical paradigm shift from teacher-centered to learner-centered education, the needs and attentions 
of critical thinking have enlarged at the center of higher education (Barab, Kling, & Gray, 2004; Duffy & 
Kirkley, 2004; Reigeluth, 1999). 

How can we promote student critical thinking to achieve the ultimate goal of higher education as 
well as satisfy the needs for the work place? A number of educational researchers and practitioners have 
discussed that online group discussion through computer-mediated communication (CMC) can serve as a 
core to foster critical thinking skills, while simultaneously promote quantity and quality of group interactions 
(Angeli, Valanides, & Bonk, 2003; Archer et ah, 2001; Garrison, Anderson, & Archer, 2000; Garrison & 
Cleveland-lnnes, 2005; Henri, 1992; Hiltz, Coppola, Rotter, & Turoff, 2000; Jeong, 2001; Johnson, Johnson, 
& Stanne, 1986; Khan, 1997; Nachmias, Mioduse, Oren, & Ram, 2000; Newman, Webb, & Cochrane, 1995). 
In addition, the guidance of facilitators in online collaborative discourse has been emphasized as a helpful 
instructional strategy to cultivate students’ critical thinking (Anderson, Rourke, Carrison, & Archer, 2001; 
Collison, Elbaum, Haavind, & Tinker, 2000; Duffy, Dueber, & Hawley, 1998; Lee, 2003). 

Therefore, as a first step in designing the effective online facilitation strategies for critical thinking, 
this paper will identify the meanings and aspects of critical thinking and compare the current four online 
facilitation guidelines for critical thinking. 

The Meanings and Aspects of Critical Thinking to Promote 
Definitions of Critical Thinking 

Critical thinking in literature has been defined in various ways associated with critical inquiry, 
critical reasoning, and cognitive presence (Anderson et ah, 2001; Angeli et ah, 2003; Angelo, 1995; Archer 
et ah, 2001; Brookfield, 1987; Duffy et ah, 1998; Lee, 2003; Pithers & Soden, 2000). 

Critical thinking can be defined as “the intellectually disciplined process of actively and skillfully 
conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or 
generated by, observation, experience, reflection, reasoning or communication, as a guide to belief and 
action” (Paul & Elder, 2001, p. 371). From another perspective, Duffy, Dueber, and Hawley (1998) 
associated critical thinking with an inquiry process. They argued that a learner inductively or deductively 
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solves his or her puzzlement through the inquiry process of generating a hypothesis, gathering and evaluating 
data, considering alternatives, and resolving for a rational solution. On some occasions, critical thinking and 
cognitive presence have been used synonymously. Garrison, Anderson, and Archer (2001) defined cognitive 
presence as “the extent to which learners are able to construct and confirm meaning through sustained 
discourse in a critical community of inquiry” (p. 1). In this way, various definitions and terms referred to as 
critical thinking to describe the focus of specific interests have coexisted in literature. 

Nevertheless, there is a general consensus to distinguish critical thinking beyond this diversity. It is 
that inquiry is the key notion of critical thinking (Hagedorn et al., 1999; Henri, 1992; Savery & Duffy, 1995; 
Tsui, 2000). Based on the inquiry, critical thinking is explained as (a) a productive and positive activity, (b) a 
process, not an outcome, (c) a manifestation depending on contexts, (d) an event triggered by positive and 
negative stimuli, and (e) an emotive and rational response (Brookfield, 1987). 

Components of Critical Thinking in Online Learning Environments 

There are also many different models to explain the components of critical thinking in online 
conferencing. Interestingly enough, most of these models were originally developed to evaluate critical 
thinking levels as content analysis tools (Garrison et ah, 2000 and 2003; Henri, 1992; Newman, Webb, & 
Cochrane, 1995). 

First of all, Henri’s (1992) critical reasoning skill stages in the problem-solving process are 
considered as a fundamental model. As critical thinking indicators, Henri’s conceptual framework includes 
the following four key stages: (a) elementary clarification observing or studying a problem, identifying its 
elements, and observing their linkages; (b) in-depth clarification analyzing a problem to understand its 
underlying values, beliefs and assumptions; (c) judgment making decisions, evaluations, and criticism; and 
(d) strategies for application of solution following a choice or decision (Jeong, 2001, p. 12). 

However, in terms of visualizing dynamic relationships of key critical thinking components in an 
online learning community, Garrison and his colleagues’ Practical Inquiry model (2000, 2001, and 2003) is 
regarded as a representative theoretical framework (see Figure 1). Particularly, the notion of critical thinking 
in this Practical Inquiry model refers to cognitive presence, one of the three overlapping elements in the 
Community of Inquiry (i.e., cognitive presence, social presence, teaching presence', see Figure 2). We should 
notice that the relationships of overlapping parts among cognitive, social, and teaching presences can 
influence facilitation strategies for critical thinking in many different ways. This issue will be discussed more 
in the following section. 



Figure 1. Practical Inquiry model (Garrison et al,, 2001, p .99) 
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Figure 2. Elements of an educational experience in a Community of Inquiry (Garrison et al., 2000, p .88) 
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The Practical Inquiry model well represents the conceptual relations of critical thinking components 
in the inquiry process, including four phases (i.e., a triggering event, exploration, integration, and 
resolution). First of all, an issue or problem arises as a triggering event, the first phase of inquiry (see Table 
1). After that, brain-storming and exchange of information occur in exploration. Based on the divergent 
phase of exploration, learners construct shared meaning in integration. In the last phase, resolution, learners 
apply the issue or problem that emerged in the triggering event (Archer et al., 2001; Garrison et al., 2001). 



Table 1. Four phases of critical thinking (Garrison et al., 2001) 



Events 


Indicators 


Processes 


Examples 


1. Triggering 

event: 

Evocative 


1 . 1 Recognizing the 
problem 


Presenting background information 
that culminates in a question. 


It has been argued 
that the only way to 
deliver effective 
distance education is 
through a systems 
approach. However, 
this approach is 
rarely used. Why do 
you think that is? 


1 .2 Sense of puzzlement 


Asking questions 

Messages that take discussion in a 

new direction 


2. Exploration: 
Tentative 


2.1 Divergence: within 
the online community 


Unsubstantiated contradiction of 
previous ideas 


One reason I think it 
is seldom used is that 
it is too complicated 
to gain cooperation. 
Another may be the 
mind-sets of those in 
charge of changing 
practices. 


2.2 Divergence: within a 
single message 


Many different ideas/themes 
presented in one message 


2.3 Information 
exchange 


Personal narratives/descriptions/ 
facts (not used as evidence to 
support a conclusion) 


2.4 Suggestions for 
consideration 


Author explicitly characterizes 
message as explorations (e.g., 
“Does that seem about right?”, 
“Am I way off the mark?”) 
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2.5 Brainstorming 


Adds to established points but does 
not systematically defend/justify/ 
develop addition 






2.6 Leaps to conclusions 


Offers unsupported opinions 






3.1 Convergence among 
group members 


Reference to previous message 
followed by substantiated 
agreement (e.g., “I agree 
because...”) 

Building on, adding to others’ 
ideas 


We also had trouble 
getting cooperation. 
Often the use of new 
tools requires new 
organizational 
structures. 


3. Integration: 
Provisional 


3.2 Convergence within 
a single message 


Justified, developed, defensible, 
yet tentative hypotheses. 


We addressed these 




3.3 Connecting ideas, 
synthesis 


Integrating information from 
various sources — text book, 
articles, personal experience 


issues when we 
implemented a 
systems approach, 




3.4 Creating solutions 


Explicit characterization of 
message as a solution by 
participant 


and 1 think that’s why 
we were successful. 


4. Resolution: 


4.1 Vicarious 
application to real world 


Non coded 


How we solved this 
problem was...? 


Committed 


4.2 Defending solutions 


Non coded 





As mentioned above, because of the original purpose, this practical inquiry model can be used not 
only for understanding the phases of the critical thinking process but also for analyzing the quality of critical 
thinking in online discourse. Generally in the analysis of critical thinking, integration and resolution are 
considered as vital aspects related to high quality critical thinking, rather than a triggering event and 
exploration (Garrison et al., 2000). This is because integration and resolution deal with synthesizing and 
application. Therefore, in order to design effective facilitation strategies to foster critical thinking, colleges 
should pay more attention to the integration and resolution phases. 

Online Facilitation Strategies to Promote Critical Thinking 

Learners’ critical thinking skills can be supported and enhanced through the guidance of facilitators 
in online collaborative learning situations such as instructional activities to scaffold student thinking process 
(Ahern, Peck, & Laycock, 1992; Brookfield, 1987; Garrison et ah, 2005; Jonassen, 1996; Salmon, 2000; 
Sharma & Hannafin, 2004; Zhu, 1998). Therefore, in this section, four online mentoring guidelines designed 
by Collison, Elbaum, Haavind, and Tinker (2000), Duffy et ah (1998), Angeli et ah (2003), and Anderson et 
ah (2001), will be described and compared in terms of general online facilitation strategies and specific 
examples. This review will be helpful to provide better understanding about online discourse facilitations and 
to create effective facilitation strategies based on the context of each institution. 

These four online facilitation guidelines have been selected because their goals, values, and 
components are clearly identified (Reigeluth, 1999), and their importance and applications can be 
considerable in the wide range of online learning. All of the four online discourse facilitation guidelines are 
based on asynchronous message transmission and text-based communication conferencing systems. 

Collison and His Colleagues’ Critical Thinking Facilitation Strategies in Online Learning 

Collison, Elbaum, Haavind, and Tinker (2000) considered critical thinking as an individual learner’s 
internal process, and they proposed detailed facilitation strategies to mediate information and promote 
learners’ critical thinking in online learning environments (see Table 2). Basically, their facilitation strategies 
suggest that learners’ critical thinking should be integrated through two main instructional strategies: (a) 
sharpening the focus and (b) deepening the dialogue. By sharpening the focus, facilitators can clarify the 
ideas and bring a common understanding among learners. This facilitating strategy highlights relevant ideas 
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and key contributions, brings coherence, and pushes the dialogue forward. In the strategy of deepening the 
dialogue, facilitators can extend learners’ thoughts by articulating more in-depth discussions on the common 
ground. Also in the deepening the dialogue strategy, facilitators can build on common ground where 
participants examine their own beliefs and assumptions, reflecting on perturbations to build new levels of 
understanding (Collison et al„ 2000). 



Table 2. Critical thinking facilitation strategies, adopted and modified from Collison et al. (2000) 



General strategies 


Specific Strategies 


Sharpening 
the focus 


Identifying direction 


• Selectively highlight or paraphrase relevant discussion, 
items in order to refocus or redirect conversation, possibly 
weaving several discussion threads or ideas to provide a 
new focus. 


Sorting ideas for 
relevance 


• Ask students to classify or form comparisons. 

• Alternatively, if needed, call attention to sorting of ideas, 
making public the sorting mechanism, to focus on 
relevance and importance. 

• Highlight “hidden gems” in postings to bring them out of 
obscurity. 


Focusing on key points 


• Ask them to summarize or synthesize. 

• If necessary, eventually highlight key contributions, 
essential concepts, and connections so far. 

• Bring to light potential gaps or tensions. 

• Provide big picture, but do not summarize in detail or infer 
future direction — push participants to draw these 
inferences and assessments by themselves. 

• Ask them to evaluate the strength of their ideas, seek 
judgments and assessments, and eventually reach 
consensus. 


Deepening 
the dialogue 


Full-spectrum 

questioning 


• Use a wide questioning approach (e.g., who, what, when, 
where, why) to push participants to examine their own 
personal, or collective, thoughts and beliefs. 

• Push them to go beyond by asking “so what?” in a specific 
context 

• Ask them to consider other perspectives that they may not 
have thought about before. 

• Ask them to clarify or elaborate their ideas. 

• Ask them explore their assumptions and sources and 
provide a rationale or examples for their ideas. 

• Ask questions to identify cause and effects/outcomes. 

• Ask the team to solve discrepancies in the ideas. 

• Ask questions considering appropriate action or inquiry 
especially if the discussion is stuck. 


Making connections 


• Stretch the participant’s imagination or conceptual frames 
to consider obscure but essential similarities. 

• Move beyond the barriers of previously held beliefs or 
assumptions that may block these connections across 
contexts or at deeper levels. 


Honoring multiple 
perspectives 


• Lay out the landscape of different views present in the 
discussion. 

• This is usually the last stage before the group completes 
their final task. 



Duffy and His Colleagues’ Critical Thinking Facilitation in Electronic Conferencing Systems 
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On the other hand, Duffy, Dueber, and Hawley (1998), who assumed that critical thinking is 
collaborative inquiry, argued that high critical thinking involves building a strong argument with its essential 
elements (i.e., hypotheses, counter-arguments, and evidence). For this reason, facilitators should support 
increasing (a) the quality of an individual learner’s analysis of the problem, (b) the quality of the counter- 
arguments, and (c) the quality of the evidence in the inquiry’ process. 



Table 3. Critical thinking facilitation, adopted and modified from Duffy et al. (1998) 



General strategies 


Specific examples 


Improving the 
quality of the 
analysis 


Analyzing problems 


• What seems to be standing out in all of this? 


Identifying evidence 


• Do I know of evidence to support or refute that idea? 

• What evidence should I seek? 


Comparing alternative 
hypotheses 


• Can we approach this problem another way? 


Summarizing discussion 


• Can I organize what we have done so far? 

• What do we know so far? 


Considering 
implications of 
proposed solutions 


• What do we have to do next? 

• How does that relate to the problem/solution? 


Encouraging 
activities in 
conversation 


Monitoring 

understanding 


• Do I understand the terms that were used? 


Asking questions 


• How does that fit into the discussion? 


Developing 
order and 
focus of 
discussion 


Developing order of 
discussion 


• Where are we in relation to developing a solution? 


Developing focus of 
discussion 


• What are the key ideas and issues we have been talking 
about? 



In addition, Duffy et al. indicated that there are two distinct types of interactions in the collaborative 
inquiry, issue-based discussion and conversation, and emphasized that not only issue-based discussion (i.e., 
critical reasoning discussion) but also conversation (i.e., general group discussion) must be supported to 
foster critical thinking in the conferencing systems. 

Angeli and Her Colleagues’ Electronic Mentoring in a Web-Based Conferencing System 

As another set of cognitive facilitation strategies to increase the quality of electronic conferencing 
on the web, Angeli, Valanides, and Bonk (2003) suggested twelve online mentoring forms according to (a) 
high-level mentoring, (b) low-level mentoring, and (c) management, modified from Bonk and King (1998)’s 
mentoring model (see Table 3). In the high-level mentoring strategies, cognitive task structuring, pushing to 
explore, cognitive elaboration/explanations, fostering reflection/self-awareness, and encouraging articulation 
were applied. Instead of providing ready answers to students, the facilitators guided them to reflect their 
thinking more deeply from others’ perspectives. For low-level mentoring, six strategies were considered: 
social (and cognitive) acknowledgement, general advice/scaffolding/suggestion, feedback, direct instruction, 
questioning, and modeling/examples. For management, mentoring through private e-mail or discussion was 
proposed. 



Table 4. Electronic mentoring, adopted from Angeli, Valanides, & Bonk (2003, p. 34) 



General strategies 


Specific examples 


High level 
mentoring 


Cognitive task 
structuring 


• Y ou know, the task asks you to . . . 

• Ok, now summarize the peer responses you have received. 

• How might your textbook authors have solved this case? 


Push to explore 


• You might want to write to Dr XYZ for. . . 



• You might want to do an ERIC search on this topic. . . 
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• Perhaps there is a Web site that addresses this topic. . . 


Cognitive elaboration 
/explanations 


• Provide more information here that explains your rationale. 

• Please clarify what you mean by. . . 

• I’m just not sure what you mean by . . . 

• Please evaluate this solution a little more carefully. 


Foster reflection/ 
self-awareness 


• Restate again what the teacher did. . . 

• How have you seen this before? 

• When you took over this class, what was the first thing you 
did? 

• Describe how your teaching philosophy will vary from 
this... 

• How might an expert teacher handle this situation? 


Encourage articulation 


• What was the problem-solving process the teacher faced 
here? 

• Does anyone have a counterpoint or alternative to this 
situation? 

• Can someone give me three good reasons why. ... 

• It still seems like something is missing here. I just can’t put 
my finger on it. 


Low level 
mentoring 


Social (and cognitive) 
acknowledgment 


• Hello.... 

• I agree with everything said so far. ... 

• Wow, what a case. 

• This case certainly has provoked a lot of discussion. . . 

• Glad you could join us. 


General advice/ 

scaffolding/ 

suggestion 


• If I were in her shoes, I would. . . 

• Perhaps I would think twice about. . . 

• I know that I would first. . . 

• How totally ridiculous this all is: certainly the teacher 
should be able to provide some. . . 


Feedback 


• That shows real insight into. . . 

• Are you sure you have considered. . . 


Direct instruction 


• I think in class we mentioned that. . . 

• Chapter X talks about. . . 

• Remember back to be first week of the semester, when we 
went over X, which indicated that. . . 


Questioning 


• What is the name of the concept? 

• Another reason for this might be. . . 

• What else might be important here? 

• Who can tell me...? 

• How might the teacher. ..? 

• What is the real problem here? 


Modeling/examples 


• I think that I solved this sort of problem once when I. . . 

• Remember that video we saw on X, wherein Y decided to. . . 

• Doesn’t X give insight into this problem in case Z, when 
he/she said... 


Management 


Through private e-mail 
or discussion 


• Don’t just criticize. . . please be sincere when you respond to 
your peers. 

• If you had put your case in on time, you would have gotten 
more feedback. 

• If you do this again, we will have to take away your 
privileges. 



Anderson and His Colleagues’ Teaching Presence in a Computer Conferencing Context 
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Based on Garrison et al. (2000)’s critical thinking and practical inquiry model, Anderson and his 
colleagues (2001) developed a tool to assess teaching presence in online computer conferencing, which can 
be used as a guideline for facilitating computer conferencing (see Table 5). Depending on the role of 
facilitating discourse, the concept of teaching presence divides into three categories: (a) direct instruction, (b) 
facilitating discourse, and (c) design and organization. Nevertheless, these notions also imply the 
relationships of cognitive, social, and teaching presences, which are three key elements in the community of 
inquiry model as mentioned earlier (see Figure 1). Direct instruction among the teaching presence 
characteristics is relevant to cognitive presence, which refers to critical thinking inquiry. Facilitation 
discourse is more related to social presence 1 and design and organization is associated with management 
issues. Despite these conceptual distinctions, all three teaching presence characteristics contain overlapped 
and shared parts within the community of inquiry model due to the nature of the discourse. 



Table 5. Teaching presence facilitation strategies, adopted from Anderson et al. (2001) 



General strategies 


Specific examples 


Direct 

instruction 


Presenting 

content/questions 


• Bates says. . .what do you think? 


Focusing the discussion 
on specific issues 


• I think that's a dead end. I would ask you to consider. . . 


Summarizing the 
discussion 


• The original question was . . .Joe said. . .Mary said. . .we 
concluded that. . .We still haven't addressed. . . 


Confirming understanding 
through assessment and 
explanatory feedback. 


• You're close, but you didn't account for. . . This is important 
because... 


Diagnosing 

misconceptions 


• Remember, Bates is speaking from an administrative 
perspective, so be careful when you say. . . 


Injecting knowledge from 
diverse sources (e.g., 
textbook, articles, internet, 
personal experiences; 
pointers to resources) 


• I was at a conference with Bates once, and he said. . . 

• You can find the proceedings from the conference at 
http://www. . . . 


Responding to technical 
concerns 


• If you want to include a hyperlink. . . 


Facilitating 

discourse 


Identifying areas of 
agreement/disagreement 


• Joe, Mary has provided a compelling counter-example to 
your hypothesis. Would you care to respond? 


Seeking to reach 
consensus/understanding 


• I think Joe and Mary are saying essentially the same thing. 


Encouraging, 
acknowledging, or 
reinforcing student 
contributions 


• Thank you for your insightful comments. 


Setting climate for 
learning 


• Don't feel self-conscious about “thinking out loud” on the 
forum. This is a place to try out ideas after all. 


Drawing in participants, 
prompting discussion 


• Any thoughts on this issue? 

• Anyone care to comment? 


Assess the efficacy of the 
process 


• I think we're getting a little off track here. 


Design and 
organization 


Setting curriculum 


• This week we will be discussing. . . 


Designing methods 


• I am going to divide you into groups, and you will debate. . . 


Establishing time 


• Please post a message by Friday. . . 



1 The authors identified that facilitation discourse is not completely related to the pure social aspect (e.g., 
chatting, coffee room), unlike the notion of “social” in the previous researches of their study. 



206 








parameters 






Utilizing medium 
effectively 


• Try to address issues that others have raised when you post. 




Establishing netiquette 


• Keep your messages short. 



Comparisons of Four Online Facilitation Guidelines for Critical Thinking 

So far, each of the four online facilitation guidelines for critical thinking has been reviewed 
regarding the general strategies and specific examples (Collision et al., 2000; Duffy et al., 1998; Angeli et al., 
2003; Anderson et al., 2001). Flowever, in order to create ideal facilitation strategies for promoting critical 
thinking, it is necessary to compare four online facilitation guidelines in a single conceptual framework. This 
comparison will allow researchers to discover the strengths and weaknesses of each facilitation guideline. As 
the conceptual framework for this comparison, Angeli et al.’s mentoring levels (2003) and Garrison et al.’s 
critical thinking phases (2000) were adopted. Only main categories of the general facilitation strategies in the 
four online facilitation guidelines were reorganized to compare the similarities and differences (see Table 6). 



Table 6. Online facilitation strategies for critical thinking, adopted and modified from Anderson et al. 
(2001), Angeli et al. (2003), Collison et al. (2000), and Duffy et al. (1998) 



Critical thinking 


Collison et al. 
(2000) 


Duffy et al. 
(1998) 


Angeli et al. 
(2003) 


Anderson et al. 
(2001) 


Level 


Phase 


High level 
critical 
thinking 
facilitation 


Triggering 

event 


Identifying 

direction 


Analyzing 

problems 


Cognitive task 
structuring 


Presenting 

content/questions 


Exploration 


Full-spectrum 

questioning 


Asking questions 


Push to explore 


Diagnosing 

misconceptions 


Monitoring 

understanding 


Confirming 

understanding 

through 

assessment and 

explanatory 

feedback. 


Summarizing 

discussion 


Summarizing the 
discussion 


Identifying 

evidence 


Cognitive 

elaboration/ 

explanations 


Injecting 
knowledge from 
diverse sources 


Soiling ideas 
for relevance 


Developing order 
of discussion 






Focusing on 
key points 


Developing focus 
of discussion 


Focusing the 
discussion on 
specific issues 


Integration 


Making 

connections 


Considering 
implications of 
proposed solutions 


Encouraging 
articulation and 
fostering 
reflection/self- 
awareness 
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Flonoring 

multiple 

perspectives 


Comparing 

alternative 

hypotheses 




Identifying areas 
of agreement/ 
disagreement 

Seeking to reach 

consensus/underst 

anding 


Resolution 










Low level 
critical 
thinking 
facilitation 








General 

advice/ scaffolding/ 

suggestion 

Direct instruction 

Modeling/ 

examples 

Questioning 

Feedback 




Social (and 

cognitive) 

acknowledgment 


Encouraging, 
acknowledging, or 
reinforcing student 
contributions 




Setting climate for 

learning 

Drawing in 

participants, 

prompting 

discussion 


Non- 

critical 

thinking 

facilitation 








Through private 
e-mail or 
discussion 


Setting curriculum 

Designing 

methods 

Establishing time 
parameters 
Utilizing medium 
effectively 
Establishing 
netiquette 
Assessing the 
efficacy of the 
process 
Responding to 
technical concerns 



Based on the comparison of online instructional strategies for critical thinking, there are four major 
findings. First of all, in terms of conceptual scopes and boundaries, two facilitation guidelines did not 
consider certain aspects of critical thinking, such as social and organizational aspects. Due to the nature of 
CMC as shown in the community of inquiry model (see Figure 2), all three dimensions in online discourse 
(i.e. cognitive, social, and teaching presences) somewhat overlap and influence each other. For this reason, 
several researchers, such as Paulsen (1995) and Mason (1991), divided the online educational moderators’ 
role into three major responsibilities: intellectual, social, and organizational. Flowever, Collision et al. and 
Duffy et al.’s facilitation strategies excluded social and organizational aspects, which are relatively less 
related to the notion of critical thinking. Meanwhile, Angeli et al. and Anderson et al. involved some of these 
aspects, such as low-level mentoring, social presence (i.e., setting climate in Figure 2), and pure teaching 
presence (i.e., structure/process in Figure 2). Collison et al. and Duffy et al. might want to present only key 
strategies directly related to promoting critical thinking. Flowever, it would be necessary to consider the 
dynamic relationships of cognitive and social presences in online discourse to foster critical thinking (e.g., 
social and cognitive acknowledgement for encouraging discussion activities of Angeli et al.). 
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Second, in terms of the components and ratios, there are similar patterns between the reorganized 
facilitation strategies in this paper and the content analysis results of critical thinking in Garrison et al. 
(2001)’s study. Overall, reorganized facilitation strategies (see Table 6) cover three phases (i.e., a triggering 
event, exploration, and integration) among four critical thinking phases. Also, interestingly, the ratios of 
these reorganized facilitation strategies in terms of critical thinking phases are similar to those of the results 
in Garrison et al.’s content analysis of an online discussion (2001). For example, exploration (42%), 
integration (13%), a triggering event (8%), and resolution (4%), excluding others related to social presence 
(33%), were shown in their analysis results. Meanwhile, exploration (16), integration (8), and a triggering 
event (4) were in the order of the total numbers of reorganized general facilitation strategies on Table 6. 
However, any single strategy of the four facilitation guideline sets will not fit into resolution. As shown in 
Garrison et al.’s analysis result, resolution may not be often observed in real online discourse. In fact. 
Garrison and his colleagues tried to explain this interesting result related to resolution considering three 
potential reasons associated with the goals, the medium, and the practical inquiry model. Despite the lack of 
frequency, resolution is still considered as the most important event among the four phases in terms of 
judging the quality of critical thinking in discourse as well as completing the critical inquiry process. 

Third, based on the assumption of critical thinking related to inquiry, several similarities and 
differences are shown in terms of the focus of facilitation strategies. For example, Duffy et al. and Anderson 
et al., whose facilitations were based on a strong notion of collaborative inquiry, presented relatively more 
detailed strategies in exploration. Meanwhile, Angeli et al. proposed several direct instructional guidelines 
(e.g., direct instruction, modeling/examples, feedback) as low-level mentoring. On the other hand, interesting 
differences between Duffy et al. and Anderson et al.’s strategies were shown. Both of them emphasized the 
importance of encouraging group discussion (i.e. encouraging activities in conversation in Duffy et al. and 
facilitating discourse in Anderson et al.; see Tables 3 and 5). However, the general facilitation strategies 
selected to promote group discussion in one study contrasted with the other. Duffy et al. focused on cognitive 
reasoning (e.g., asking questions, monitoring understanding), while Anderson et al. added social presence 
strategies (e.g., encouraging, acknowledging contributions, setting climate for learning). 

Last, several facilitation strategies were under inappropriate categories in their original guidance. 

For instance, responding to technical concerns was initially placed in the cognitive presence category called 
direct instruction (see Table 5). However, because the focus of responding to technical concerns is not 
related to the cognitive aspect in discourse, it should be under the non-critical thinking facilitation category 
as shown on Table 6. In this way, identifying areas of agreement and disagreement and seeking to reach 
consensus/understanding, which originally belonged to the social presence category called facilitating 
discourse (see Table 5), were moved to the integration of high level critical thinking facilitation. 

How to Promote Critical Thinking: Conclusion and Caveats 

Critical thinking becomes a prime goal of postsecondary education targeted for future mid-career 
knowledge workers, concurring with the needs in the work place. Now the importance of critical thinking has 
been emphasized in higher education more than ever. Therefore, as the vital aspects of critical thinking, four 
events of cognitive presence (i.e., a triggering event, exploration, integration, resolution) from Garrison et 
al.’s community of inquiry (2000) were reviewed in this study. The online facilitation guidelines of Collison 
et al. (2000), Duffy et al. (1998), Angeli et al. (2003), and Anderson et al. (2001) were described and 
reorganized to compare their similarities and differences at the levels and phases of critical thinking. 

However, in the comparison of these facilitation guidelines, several critical points are founded that 
cannot be easily dismissed. To overcome these weak points, the following three suggestions can be 
considered. 

First, in terms of the conceptual scopes and boundaries of critical thinking, the relationships 
between social presence and cognitive presence should be more carefully considered in facilitation 
guidelines. This is because critical thinking includes both emotive and rational aspects at the same time 
(Brookfield, 1987; Garrison et al., 2000). For instance, in a case of encouraging participation in discussion, 
providing facilitation strategies emphasizing social aspects may be more effective. 

Second, missing strategies in several subcomponents of the phases should be designed, regarding 
the relations between the facilitation strategies and critical thinking phases. For example, resolution and 
integration are crucial events in order to evaluate the quality of online discourse during the critical inquiry 
process (Garrison et al., 2000; Duffy et al., 1998). Nonetheless, Garrison et al. (2000) and Anderson et al. 
(2001)’s strategies did not provide specific facilitation guidance for the resolution event in teaching 
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presence. In fact, some critical thinking steps are more personal reasoning processes, so facilitation is not 
much help for these critical thinking steps. However, the resolution event belongs to the area of discourse in 
the shared world, according to Garrison et al. (2000)’s practical inquiry model (see Figure 2). For this reason, 
it is obvious that facilitators’ mentoring in the resolution event is more essential to the learners’ inquiry 
processes. 

Last, facilitator’s intervention in discourse should be further investigated to figure out ideal 
intervention levels. Due to the nature of the inquiry process as well as the personal style of the facilitator, to 
control intervention degrees in online mentoring is not easy, especially for novice facilitators. Thus, 
considering various inquiry situations and facilitators’ personal values, more detail facilitation guidelines 
should be designed (Reigeluth, 1999). 

The four facilitation guidelines discussed in this paper can not be the best for all colleges, even if 
they are modified according to the above suggestions, because the contexts of institutions are different. Thus, 
each college should develop proper facilitation strategies for their own institutions and students to achieve 
the ultimate goal of critical thinking. In the same way, the methods of mentoring facilitators should be 
designed, considering unique situations of organizations and faculties. Furthermore, the colleges should 
examine the effects of their own facilitation strategies and mentoring programs through design-based 
research approaches and continuously modify them based on the findings (Barab, Kling, & Gray, 2004; 
Reigeluth & Frick, 1999; Richey, Klein, & Nelson, 2004; Winn 2002). 

References 

Ahern, T. C., Peck, K., & Laycock, M. (1992). The effects of teacher discourse in computer-mediated 
discussion. Journal of Educational Computing Research, 5(3), 291 - 309. 

Anderson, T., Rourke, L., Carrison, D. R., & Archer, W. (2001). Assessing teaching presence in a computer 
conferencing context. Journal of Asynchronous Learning Networks, 5(2), 1-17. 

Angeli, C., Valanides, N., & Bonk, C. J. (2003). Communication in a web-based conferencing system: The 
quality of computer-mediated interaction. British Journal of Educational Technology’, 34( 1), 31-43. 
Angelo, T. A. (1995). Classroom assessment for critical thinking. Teaching of Psychology’, 22(1), 6-7. 

Archer, W., Garrison, R. D., Anderson, T., & Rourke, L. (2001). A framework for analysing critical thinking 
in computer conferences. In P. Dillenbourg, A. Eurelings & K. Hakkarainen (Eds.), European 
perspectives on computer-supported collaborative learning. Proceedings of the First European 
Conference on CSCL. Maastricht, Netherlands: McLuhan Institute, University of Maastricht. 
Association of American Colleges. (1985). Integrity’ in the college curriculum: A report to the academic 
community. Washington, DC: Association of American Colleges. 

Barab, S. A., Kling, R., & Gray, J. H. (Eds.). (2004). Designing for virtual communities in the service of 
learning. Cambridge, NY: Cambridge University Press. 

Bennett, N., Dunne, E., & Carre, C. (1999). Patterns of core and generic skill provision in higher education. 
Higher Education, 37, 71-93. 

Bonk, C. J., & Dennen, V. (2003). Frameworks for research, design, benchmarks, training, and pedagogy in 
Web-based distance education. In M. G. M. W. Anderson (Ed.), Handbook of distance education 
(pp. 331-348). Mahwah, NJ: Erlbaum. 

Bonk, C. J., & King, K. S. (Eds.). (1998). Electronic collaborators: Learner-centered technologies for 
literacy, apprenticeship, and discourse. Mahwah, NJ: Erlbaum. 

Brookfield, S. D. (1987). Developing critical thinkers: Challenging adults to explore alternative ways of 
thinking and acting. San Francisco: Jossey-Bass. 

Collison, G., Elbaum, B., Haavind, S., & Tinker, R. (2000). Facilitating online learning: Effective strategies 
for moderators. Madison, WI: Atwood. 

Duffy, T. M., Dueber, B., & Hawley, C. L. (1998). Critical thinking in a distributed environment: A 
pedagogical case for the design of conferencing systems. In C. J. Bonk & K. S. King (Eds.), 
Electronic collaborators: Learner-centered technologies for literacy, apprenticeship, and discourse 
(pp. 51-78). Mahwah, NJ: Erlbaum. 

Duffy, T. M., & Kirkley, J. R. (Eds.). (2004). Learner-centered theory’ and practice in distance education: 
Cases from higher education. Mahwah, NJ: Erlbaum. 

Facione, P. A., Facione, N. C., & Giancarlo, C. A. (2000). The disposition toward critical thinking: Its 
character, measurement, and relationship to critical thinking skill. Informal Logic, 20(1), 61-84. 



210 




Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer 
conferencing in higher education. The Internet and Higher Education, 2(3), 1-19. 

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer 
conferencing in distance education. American Journal of Distance Education, 75(1), 7-23. 

Garrison, D. R., Anderson, T., & Archer, W. (2003). A theory of critical inquiry in online distance education. 
In M. G. Moore & W. G. Anderson (Eds.), Handbook of Distance Education (pp. 113-127). 
Mahwah, NJ: Erlbaum. 

Garrison, D. R., & Cleveland-lnnes, M. (2005). Facilitating cognitive presence in online learning: Interaction 
is not enough. American Educational Research Journal, 79(3), 133-148. 

Hagedorn, L. S., Pascarella, E. T., Edison, M„ Braxton, J., Nora, A., & Terenzini, P. T. (1999). Institutional 
context and the development of critical thinking: A research note. The Review of Higher Education, 
22(3), 265-285. 

Harvey, L., Moon, S., & Geal, V. (1997). Graduates ’ work: Organizational change and students ’ attributes. 
Birmingham: University of Central England, Center for Research into Quality. 

Henri, F. (1992). Computer conferencing and content analysis. In C. O'Malley (Ed.), Computer supported 
collaborative learning. Heidelberg, Germany: Springer- Verlag. 

Hiltz, S. R., Coppola, N., Rotter, N., & Turoff, M. (2000). Measuring the importance of collaborative 
learning for the effectiveness of ALN : A multi-measure, multi-method approach. Journal of 
Asynchronous Learning Networks, 4(2), 103-125. 

Jeong, A. C. (2001). Supporting critical thinking with group discussion on threaded bulletin boards: An 
analysis of group interaction. Madison: University of Wisconsin. 

Johnson, R. T., Johnson, D. W., & Stanne, M. B. (1986). Comparison of computer-assisted cooperative, 

competitive, and individualistic learning. American Educational Research Journal, 23(3), 382-392. 

Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking. Englewood Cliffs, NJ: 
Merrill. 

Khan, B. H. (Ed.). (1997). Web-based instruction. Englewood Cliffs, NJ: Educational Technology. 

Lee, M. (2003). Impacts of cognitive structuring methods on students ’ critical thinking enhancement in on- 
line collaborative learning. Chicago: University of Illinois. 

Mason, R. (1991). Moderating educational computer conferencing. DEOSNEWS, 7(19). 

Nachmias, R., Mioduse, D., Oren, A., & Ram, J. (2000). Web-supported emergent-collaboration in higher 
education courses. Educational Technology > & Society’, 3(3), 94-104. 

National Education Goals Panel. (1991). Executive summary’: The national education goals report, 1991: 
Building a nation of learners . Washington, DC: National Education Goals Panel. 

National Institute of Education Study Group. (1984). Involvement in learning: Realizing the potential of 

American higher education. Washington, DC: National Institute of Education, U.S. Department of 
Education. 

Newman, D. R., Webb, B., & Cochrane, C. (1995). A content analysis method to measure critical thinking in 
face-to-face and computer supported group learning, interpersonal computing and technology. An 
Electronic Journal for the 21st Century, 3(2), 56-77 . 

Paul, R., & Elder, L. (2001). Critical thinking: Tools for talking charge of your learning and your life. Upper 
Saddle River, NJ: Prentice Hall. 

Paulsen, M. P. (1995). Moderating educational computer conferences. In Z. L. Berge & M. P. Collins (Eds.), 
Computer-mediated communication and the on-line classroom in distance education. Cresskill, NJ: 
Hampton Press. 

Pithers, R. T., & Soden, R. (2000). Critical thinking in education: A review. Educational Research, 42(3), 
237-249. 

Reigeluth, C. M. (Ed.). (1999). Instructional-design theories and models: A new paradigm of instructional 
theories (Vol. 2). Mahwah, NJ: Erlbaum. 

Reigeluth, C. M., & Frick, T. W. (1999). Formative research: A methodology for creating and improving 
design theories. In C. M. Reigeluth (Ed.), Instructional-design theories and models: A new 
paradigm of instructional theories (Vol. 2). Mahwah, NJ: Erlbaum. 

Richey, R. C., Klein, J. D., & Nelson, W. A. (2004). Developmental research: Studies of instructional design 
and development. In D. H. Jonassen (Ed.), Handbook of research on educational communications 
and technology’ (2nd ed., pp. 1099-1130). Mahwah, NJ: Erlbaum. 

Salmon, G. (2000). E-moderating: The key to teaching and learning online. London: Kogan Page. 



211 




Savery, J. R., & Duffy, T. M. (1995). Problem based learning: An instructional model and its constructivist 
framework. Educational Technology, 55(5), 31-38. 

Sharma, P., & Hannafin, M. (2004). Scaffolding critical thinking in an online course: An exploratory study. 
Journal of Educational Computing Research, 31(2), 181-208. 

Slee, P. (1989). A consensus framework for higher education. In C. Ball & H. Eggins (Eds.), Higher 
education in the 1990s: New dimensions'. SRHE/OUP. 

Tsui, L. (2000). Effects of campus culture on students' critical thinking. The Review of Higher Education, 
23(4), 421-441. 

US Congress. (1994). Goals 2000: National Goals for Education Act. Washington, DC: US Government 
Printing Office. 

Winn, W. (2002). Current trends in educational technology research: The study of learning environments. 
Educational Psychology’ Review, 14(3), 331-351. 

Zhu, P. (1998). Learning and mentoring: Electronic discussion in a distance learning course. In C. J. Bonk & 
K. S. King (Eds.), Electronic collaborators: Learner-centered technologies for literacy, 
apprenticeship, and discourse (pp. 233-259). Mahwah, NJ: Erlbaum. 



212 




Learning to Think like Instructional Designers in a Kaleidoscope of Cultures: 
Using Reflective Practices in Thailand, India, and the US 



Tiffany A. Koszalka, Syracuse University - USA 
Lynn-Beth Satterly, Upstate Medical University - USA 
Radha Ganesan, QUEST Alliance - India 
Bro. Anant Prichavudhi, Assumption College, Thailand 

Abstract 

Engaging novices in reflective activities can enhance instructional design (ID) practices 
(Bellefeuille, 2006; Orland-Barak & Yinon, 2007). As part of a community of practitioners it is 
expected that instructional designers will engage with the resources and people of the community, 
develop ID competencies, and help enhance community practices. Reflective practices like 
engaging in relaxation activities prior to contemplation, drawing and writing, and peer critique 
activities can foster learning and the transfer of ideas into improved practices (Cyboran, 2005). 

Reflective practices can support newcomers in thinking about their role and the work of the 
community, developing new perspectives on ID, and sharing the meaning of their work with 
other members in a continual process of transforming ID practices. This paper describes theories, 
techniques, guidelines, and examples of reflective practices in action with ID newcomers. 

Introduction 

Becoming a member of a practice community often begins with some vague ideas of the members’ work 
(Lave & Wenger, 1998). A mental model is conceived that explains the community members’ identities, e.g., their 
goals, how they frame problems, what rules they use for engaging in the community’s work, and which tools and 
resources they choose to support problem solving (Engestrom, 1999; Kuttii, 1991; Schon, 1987). These ideas 
develop over time through learning, observation, conversation, inquiry into the practices of the community, and 
reflection. As individuals begin to make choices to join and engage in a community, other experiences (e.g., 
peripheral engagement) enlighten their journey toward, and understanding of, their role in the community’s practices. 
Engaging in guided and individual reflection can help newcomers learn about the work, rules, tools, methods, 
practices, and relationships within the community and their own roles and responsibilities as practitioners. This 
reflection both informs knowledge and competency development and encourages community building in one’s own 
context (Barry, 1994; Bellefeuille, 2006; Lave & Wenger, 1998; Orland-Barak & Yinon, 2007). Continued reflective 
practices coupled with dialogue among newcomers and old timers can bring newcomers from peripheral 
participation into a richer and fuller understanding of, and participation in, the work of the community. Ultimately 
this leads to the transformation and improvement of the community’s overall practices and the individual’s 
recognition of meaning in his or her work (Korthagen, 1990; Lave & Wenger, 1991). 

Theory Behind Community Building through Reflective Practices 

In this context, a community of practice is defined as a culture or a group of individuals who engage in a 
common tradition of work, e.g., educators, instructional designers (Levinson & Holland, 1996). The members of the 
community transmit their knowledge to newcomers through a process of creating and sharing meaningfulness and 
engaging them in peripheral practices and with resources (e.g., people, tools, methods, theories, rationale, etc), 
eventually leading into full participation in the work of the community (Lave & Wenger, 1991). The members 
engage in activities to maintain, change, and expand practices based on new ideas and knowledge brought into the 
community. Thus, some communities remain stable and interact mainly within their own community. Others evolve 
beyond their practice contexts to engage, share, and interact as interdisciplinary communities (McArdle & Ackland, 
2007). For example instructional design community members engage far beyond formal education contexts in K12, 
higher education, and continuing education, into areas such organizational design, human performance, and 
technology design and development in multiple content domains. The combination of learning, reflective practices, 
and sharing of ideas is at the root of the enculturation process within a community and how it dynamically 
transforms itself based on member responsiveness to the world (Deglau et ah, 2006; Levinson & Holland, 1996). 
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A starting point in understanding these community level changes is to consider the assumption that an 
individual’s behavior is directed by cognition. Individuals create mental structures (schema) as models of reality 
which helps them to interact with the environment and to anticipate future behaviors and actions (Korthagen, 1990). 
Prior knowledge is a key component in the learning process in which these schema are created (Bransford, 1979). 
Schema can be modified through a gradual accumulation of information (accretion), evolutionary changes in the 
way information in an existing schema is interpreted thus changes a schema (tuning), or through completely 
restructuring new information to create a new schema (Rumelhart & Norman, 1981; Vosniadou & Brewer, 1987). 
Reflection plays a significant role in helping the individual identify inconsistencies, gaps, or problems with his or 
her models and determine ways to close or reframe his or her thinking, and ultimately his or her practices (Schon, 
1987). Individual and guided reflection techniques can promote the process of restructuring an individual’s frames 
of reference, thus affecting the individual’s mental models and representations of his or her experiences (De Jong & 
Korthagen, 1989 as cited from Korthagen, 1990). Further, sharing these models within a community of practice 
supports the learning (enculturation) process and provides the vehicles with which to enhance practice. 

One of Dewey’s (1933) basic assumptions was that learning improves to the degree that it arises out of the 
process of reflection. Reflection arises because the organism detects the appearance of incompatible factors within a 
situation then develops opposed responses in an attempt to further engage in and understand the situation thereby 
constructing knowledge (schema). Knowing therefore is not a process of registration or representation, but one of 
active intervention. Knowledge is constructed, in part, through reflection, e.g., ongoing active, persistent, and 
thoughtful consideration and participation in a situation (Canning, 1991). The cycle of reflecting and learning is thus 
determined by the changes one finds satisfactory about a new situation on the whole or by the discovery of new 
features that give the situation new meaning. Reflection is important in that it encourages humans (organisms) to 
explore and implement new practices or ways of thinking (incompatible factors) into their context (situation) to 
reduce the perception of incompatibility. Such reflections may prompt individuals to face environmental constraints, 
incrementally develop new practices that lead to successful practices, and specify for themselves the relationships 
between theoretical benefits of new ways of thinking and successful practice (Collis, 1996; Ertmer, 2003). 

Another definition of learning implies becoming a different person with respect to the possibilities enabled 
by the system of relationships within a community of practice (Lave & Wenger, 1998). Newcomers are enculturated 
into a community through these processes of learning, thinking, and meaning making while they are situated within 
and engaging with the resources of the community. Newcomers inevitably participate in communities of 
practitioners by engaging in and with community activities, identities, artifacts, and practices (Bruner, 1996). 
Newcomers must be supported however as they transition from peripheral participation to full engagement. This 
support often comes in the form of apprenticeship-type activities accompanied by reflective practices (Lave & 
Wenger, 1998; Seely Brown, Collins, & Duguid P, 1996). As newcomers transition in the extent of their 
participation, their entire being (intellect, affect, behavior) is transformed through their own labor and reflections as 
they develop the knowledge and skills of their trade. More importantly they deepen self-understanding of their work 
and form meaningful and authentic relationships with those whom they interact in their everyday lives (Lave & 
Wenger, 1998; Seely Brown, Collins, & Duguid P, 1996; Seely Brown & Duguid, 1996). 

Thus, becoming a practitioner in the instructional sciences suggests becoming enculturated into the 
community of instructional designer practitioners. Through a series of educational and experiential activities, 
engagements with experienced members within the community, and reflective practices newcomers can begin to 
create more complete and accurate mental models of the community and find their place within it. Newcomers can 
also develop deep understanding of their profession and develop meaningful relationship with colleagues that can 
lead to transformations of practices within the community. Reflective practices help in this process, however are not 
as simple as looking at a situation and brainstorming solutions. Reflection is a continual process of deepening 
understanding through careful consideration of the internal and external factors that guide action. 

Reflective Practices Techniques and Guidelines 

Several different reflection techniques, both individual and guided reflections, have been found to support 
practice development (Armstrong 2007; Barry, 1994; Cyboran, 2005; Orland-Barak & Yinon, 2007; Pavlovich,, 
2007; Wagner, 2006). These include prompted guided reflection, peer sharing of personal reflections and critiques 
of work activities and products, and individual reflective writing and drawing activities. These techniques are used 
in both group learning and individual contexts, often being most transformative to a community when coupled with 
discourse among novice, mid-level, and expert community members. 
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Relaxation techniques are often coupled with reflective practice to help open the individuals mind to the 
many different possibilities beyond shallow environmental factors that impact practice by helping the individual to 
reflect honestly and holistically, reaching deep and meaningful levels of thought rather than simply being 
reactionary and focused on surface level thoughts. It is the deep understanding about our work as practitioners that 
when shared can impact and enhance the community practices as a whole. Reaching that deep level, or 
contemplative level, of reflection requires commitment and good technique. 

Reflection Traditions and Techniques 

There are several traditions or theoretical perspectives on reflective practice and accompanying techniques. 
Some are based in psychological practices and other in spiritual. Common among both is that the individual must 
learn how to engage in ways that activate previous knowledge, deep and authentic self recognition, and thinking. As 
a note, practicing and facilitating reflective practices that truly engage the individual and groups in thoughtful 
reflection can take many years of practice. Those who teach or guide reflective techniques generally participate in 
intense training in the techniques and practices, learn how to achieve deep reflective practices themselves, and learn 
how to help others in such techniques. The simplicity of guidelines and implementation examples presented here 
were ministered in practice by highly trained facilitators. A short list of additional resources, from psychology and 
faith-based traditions, on how to prepare to practice and facilitate such sessions are listed in the appendix. The 
following technique descriptions are modified excerpts from Healer ’s Art Instructors ’ Handbook Upstate Medical 
University composed by Dr. Lynn-Beth Satterly, MD (Satterly, 2005). 

Preparing to Facilitate Reflective Sessions 

The facilitator’s technique and enthusiasm for and comfort with this activity is key to its success. In order 
to authentically lead a group through a guided relaxation and reflection, the facilitator should have at least some 
practice and experience with relaxation and reflection techniques. If the facilitator is not an experienced mediator, he 
or she may wish to practice this exercise alone before leading a group. The facilitator should lead this exercise with 
a calm demeanor and a clear but smooth, soothing tone of voice. It is helpful if you can do a personal 
meditation/relaxation exercise just before you lead the group. Experienced mediators can often bring themselves into 
the appropriate reflective state when they need to, without formally going through the relaxation/meditation exercise. 

Do not confuse “soft" voice with soothing voice. Speak loudly enough so that all can hear you without 
straining but do not raise your voice or yell. Keep in mind that you should lead the group through these types of 
exercises slowly. Going too quickly is more problematic for the effectiveness of the activity than going too slowly. 

As you become more experienced, you may be able to “feel” the energy and the group and recognize whether or not 
they need to spend more or less time on a particular part of the guided relaxation. 

A guided relaxation can be done effectively with absolutely no tools other than the facilitator’s voice and 
facility with the particular relaxation/meditation script. Sometimes such simplicity is indicated and appropriate. 
However, sometimes certain supplements can be used to create ambiance and foster relaxation and reflection. Items 
such as candles and music can be helpful. With experience and knowledge of a particular group, setting and topic, 
the facilitator will develop a sense of when the simple or more enriched environment is appropriate. It makes the 
activity flow more smoothly if the group is given the crayons and paper for the drawing and writing activities or 
asked to have journaling supplies available before the guided relaxation is begun. These supplies, however, should 
not be in a place that interferes with the participants’ ability to open their hands and settle into comfortable positions. 

The relaxation/centering facilitator may wish to play soft background music, dim the lights and light 
candles. Desks and seats arranged in a semi-circle or a circle can be effective. Participants sometimes like to sit on 
the floor and can be invited to do so if the room allows or if participants have a mat, pillow or rug to sit on. If the 
environment has a lot of background noise, during the guided relaxation the facilitator can invite the participants to 
acknowledge the background noise and imagine it drifting out of their consciousness. Although probably not ideal, 
even lecture hall seats can “work" for seating during a guided relaxation. 

Guidelines for Relaxation and Reflective Sessions 

There are many ways to conduct reflection sessions with groups. The following guidelines were designed 
to engage students in reflecting on topics covered during a larger seed talk or lecture session. 
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The general script for a beginning relaxation session 

• I invite you to close your eyes or you may wish to focus on a focal point in the room. Settle into your 
chair or sit or lie in a comfortable position. Place your hands, palms up, either on your thighs, if you 
are seated, or by your side if you are lying down. If you are seated, you may wish to put both feet on 
the floor. Feel the chair against you back or feel your legs against the floor. Rest for a moment in 
awareness of these sensations. (Pause and let the group “remain” in this “place” for about 15-30 
seconds.) 

• Next, I invite you to focus on your breathing. Take several slow, deep breaths. . ..in and out, in and out. 
Imagine breathing in all that fills you with life and good energy. As you breathe in and out, imagine 
breathing out all that causes distress, stress or dis-ease. (Pause and let the group experience this for 
about 15-30 seconds.) The body is endowed with the breath, a natural calming and focal point. If you 
find yourself becoming distracted or unfocused, you can always return to your breathing. 

• Now, I’d like you to focus on your head and neck. Notice any discomfort, pain or tension. Imagine that 
discomfort, pain or tension as a current running from your head and neck, down your torso, down your 
legs, into you feet and out of your body and into the floor. Next, focus on your arms and torso, 
including your pelvis and the entire length of your back. Notice any discomfort, pain or tension. 
Imagine that discomfort, pain or tension as a current running from your arms and torso, down to your 
legs, into your feet and out of your body and into the floor. Repeat this process with your hips, legs and 
your feet. (Allow about 30 seconds each for the head and neck, arms and torso, and hips, legs and feet.) 

• Return, if you will, to your breath. Take in a few slow, deep breaths, breathing in all that fosters your 
health and wholeness. Feel the rush of air in your nose and feel your lungs expose with this life-giving 
substance. Exhale, slowly and fully. Feel the air rush out of your nostrils and feel your chest relax. 
Breathe out all that causes you distress or dis-ease. 

• (Allow 15 seconds so that the group can experience this.) 

• Now I invite you to imagine a place where you feel totally safe and grounded. Image everything about 
this place. Try to smell how it smells, try to feel the temperature of the air in this place, envision how 
this place looks and try to hear in your mind any of the background sounds of this special place. Any 
or all of these are your places of ground. Rest there for a few minutes now. Know that you can always 
bring your mind back to this ground when you need to. (Allow 30 seconds so that the group can 
experience this) 

• Now, in this place of the heart, where many believe intuitive knowledge resides, reflect upon... (The 
facilitator can whatever topic is relevant for the session such as wholeness and balance.) 

• When you are ready, take your crayons and drawing paper (or journal and pen). (The facilitator 
describes the chosen activity.) 

• When you are ready, return your consciousness to the room and open your eyes. Take a few minutes to 
settle into the present moment and into a state of daytime alertness. 

The general format of the small group reflection session is as follows: 

Pre-work activity (optional) 

• Ask students to bring a drawing or reflective text they were prompted to create during other related 
reflections with them 

• Ask students to bring objects with them which symbolizes an aspect of their wholeness, personal or 
professional identity. 

Create ambiance 

• Create focal point with placemat and candles or a table in the front or center of the room 

Ice breaking 

• Introductory exercise - Each person is invited to say their name and year of training and one thing that 
they’d like the group to know about themselves that people may not already know. At this time you 
can invite the students to talk about the object they brought and place it at a central location of the 
room. 

Time of grounding: (3-5 minutes) 

• Facilitator invites the students to come to quiet for a few moments and to consider again the topic of 
the evening 
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• Guided relaxation - Invite members to focus on their breathing or perhaps a candle or music or the 
background noise in the room until their mind is quiet and suggest that when their mind is quiet, they 
then focus on this moment and the meaning of their drawing. Invite them to rest there for a few 
minutes, closing their eyes if they wish. After a few minutes (don’t rush this, people usually enjoy it 
and need it), invite them to return their conscious awareness to the room. 

Small group dialogue 

• Discuss topic of the evening from a personal experience perspective. 

• Discuss and describe the activity from large group 

• Dialogue- Invite them to hold up their drawings and talk about them. Remind them that there are no 
right or wrong answers and that they should try to speak from their hearts as much as from their heads. 

• Wrap Up- About ten minutes before closing, invite anyone who has not yet spoken to speak if they 
wish, gently stating that it is close to the time to end the session. Invite others to speak if they wish. 

Closing Circle activity 

• Closing Circle Exercise- Introduce this activity. Each person will say their name at the sound of the 

tone or bell (You can think of whatever signal works for you; you can even softly say a word such as 

“peace” to signal that the next person should say their name.) and everyone in the room will sent 
unconditional positive regard silently to them. After thirty seconds, sound the signal again and the 
next person says their name. ... 

General format for conducting reflective drawing is as follows: 

Create ambiance 

• Create focal point with placemat and candles or a table in the front or center of the room 

• Before the students are taken into a guided relaxation technique, be sure that they each have a piece of 

drawing paper and several crayons, markers, or color pencils. 

Begin Guided Relaxation (as mentioned above): 

• Facilitator invites students to come to quiet for a few moments and to consider again the focus topic 

• Guided relaxation - Invite members to focus on their breathing or perhaps a candle or music or the 
background noise in the room until their mind is quiet and suggest that when their mind is quiet, they 
then focus on this moment and the meaning of their drawing. Invite them to rest there for a few 
minutes, closing their eyes if they wish. After a few minutes (don’t rush this, people usually enjoy it 
and need it), invite them to return their conscious awareness to the room. 

• During the last part of the guided relaxation, before the students are invited to leave their state of 
relaxation, introduce the following activity. 

Introduce drawing activity: 

• Ask the students to think about the topic related to their professional and personal identities and the 
practices discussed in class. In this reflection they should consider an aspect of their wholeness, 
something that they like about themselves, something that is part of their authentic selves that is related 
to their practices. It may be something that they have not cultivated since beginning graduate school. It 
may be something that they need to nurture and acknowledge. Ask them to draw a representation. 

• Suggest that although the students may have some words that they’d like to associate with their 
drawing they should focus on illustrations without words at this time. After a few minutes invite them 
to add words to their drawings if they find them useful and clarifying. 

• Note: if this is an activity occurring over time ask students to reflect on previous drawings and make 
changes or entirely new illustrations representing their reflections at this time. 

Conclude drawing activity: 

• Ask students to write a short narrative about how they feel about their drawing and what it represents 
to them in the context of their own person practices and those of the community in which they are 
preparing to practice, (and how it may have changed over time) 

• Offer an opportunity for students to share their thoughts and ideas about changes in their drawings 

• Finish this exercise by finishing the guided relaxation, (small group dialogue, closing circle) 
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Summary of Guidelines and techniques for Relaxation and Reflective Sessions 

Many other derivations of these techniques are possible. For example during short reflective drawing or 
writing activities students may be asked to illustrate their reflections about the topic of focus as guided by questions 
about what inspired or motivated them, how does their new knowledge or appreciation of their peers or own 
thoughts help them to better understand their new profession and place within it, or what confuses them about their 
work. Often such techniques are used in short 5 minute writing activities after lecture or discussion sessions. All of 
these activities are meant to prompt deeper, thinking individually or as a group, about our places within a 
community and how we can practice and inform the work of the community. The following stories demonstrate 
how ID community members were engaged in reflective technique to help them move more deeply into our 
community of practice and transform their smaller groups into effect practitioners of the larger ID community. 

Stories of Engaging Instructional Design Novices in Reflective Practices 

Three stories are told of engaging novices in different types of reflective practices to help enhance their 
abilities to think like instructional designers. Each story begins with a statement of the problem (context) in which 
the designers were engaged in reflective practices. The stories of the context then illustrate the critical nature of our 
work as instructional designers and of the depth of knowledge and flexibility we must have to resolve these complex 
problems and practice well within our contexts. The reflective techniques practiced in response to the problem and 
stories are then described. 

A Story from Work in Thailand . . . 

The Problem. Lack of technology integration practices in classroom practices. 

The Story>. New technologies were acquired at a large school in Bangkok. The school has an excellent 
reputation, high quality educators, comprehensive and interdisciplinary curriculum, and technical resources and 
personnel support. Yet, there was minimal use of these new technologies in the classrooms. Further investigation 
suggested that there were pockets of effective technology integration, but most of the educators were lacking ideas 
of how to start using technology resources within their current pedagogical content knowledge schemes. The 
educators had technical proficiencies and some basic models guiding technology integration practices, yet were 
lacking experiences of how to fully infuse technology resources into their classroom practices. The school 
administer requested that a group of educators take a leadership role in designing, developing, and implementing 
best practices of technology integration and share these new practices with their colleagues to start wide adoption of 
technology integration practices. A workshop was created to help an educator leadership group develop a better 
conception of technology integration models, reflect on how to design learning experiences with embedded 
technology resources, and most importantly develop into an effective community of technology integration leaders 
who would drive dissemination and adoption activities. The goal was to begin to create a community of leaders who 
were knowledgeable about technology integration and who trust, share, critique, collaborate, and encouragement 
engagement among its members. 

Reflective Techniques. A variety of reflective activities 
were used during the workshop sessions to help the group 
develop into a community of technology integration leaders. 

One such technique was a community building reflection 
activity that encouraged each educator to identify and share an 
object of their choosing that illustrated who they are and how 
they see themselves as a member of the school’s teaching 
community and this new leadership community. At the end of 
one of the workshop session the educators were taken through a 
series of reflective activities on technology integration models 
covered during the day. This final reflection began by engaging 
the educators in a relaxation activity where they were ultimately 
asked to think about who they were within the communities at 
the school. In their thinking they were asked to visualize objects that illustrated their thoughts and feelings. They 
were asked to bring such an object to the next session and prepare to describe how it represents them and their view 
of their role as educators and leaders. A table was setup in the middle of the room for the next session and each 
educator presented his or her item and the story behind it. Some brought small gifts from students or their own 




Figure 1. Educators share objectives that 
illustrate their reflection on who they are in 
this community of practice 
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education and shared deep thoughts about what brought them to education and their uncertainty about technology. 
Some shared items like grading books that represented broader perspectives of who they were in roles as teachers, 
and learners or report cards that also represented their roles as concerned parents. Others were very literal in that 
their items were technologies e.g., jump disk, cell phone, yet told stories of how the items represented their roles as 
communicators or keepers and sharers of information. Most also expressed in their stories the passion they have for 
being educators. Most seemed surprised at how and what their colleagues shared. See figure 1 . 

A Story from Work in India. . . 

The Problem. Poor school conditions, new technologies emerging, few quality curricular materials 
available, movement to improve the education system for the most neglected and underserved populations of 
children. 

The Story. In many parts of India the educational system for the poor is marginal at best. Over the last 
several years a major effort was undertaken to improve the government schools and increase student attendance and 
quality of education. New programs provide uniforms (clothing), food, trained teaching staff, and minimal teaching 
resources to the government schools that reach +250 million children (a ratio of 100:1 students to teachers) 
throughout India. Attendance has increased, however the quality of instructional materials and delivery of 
instruction are both still in question. Educational technologies, in the context of strong instructional design practices, 
are being investigated to help in the quest to improve the school system and create better opportunities for the poor 
to advance and become productive members in India’s society. A national workshop was designed to engage a 
cohort of 100+ content creators (instructional designer) from educational and commercial organization across India 
in activities to help them think more like instructional designers, apply sound instructional design practices and 
learning sciences in the creation of quality school materials (e-learning resources) for India’s government schools. 
Many newly develop e-learning products were showcased throughout the workshop. Upon critique many were found 
to cover basic content yet lack sound design decisions in their interface design and how they engage learners. Upon 
inquiry with the content creators (prior to and during the workshop) it was found that the they were often self-trained 
instructional designers, and although able to effectively discuss ID sciences, often they did not think beyond the 
capabilities of the technologies available and the existing curricular materials in their own practices. Consideration 
of instructional and message design principles, types of learning being addressed by the instruction, design of 
practice components, and integration of sound feedback and assessment mechanisms were often absent in their 
thinking processes. The ultimate goal of the workshop was therefore to provide guidelines on how instructional 
designers think and what factors they consider in their design practices. The group was encouraged to seek more 
understanding of the ID sciences and begin to implement personal reflective practices that engage them in thinking 
more like instructional designers ... to constantly think about the interactions among design, learning, and 
technology as well as classroom implementation issues. They were encouraged through practice and follow-up 
discussion to question each component of instructional resources as they are developed. In essence the reflective 
activities were designed to help them develop the reflective practices that are the hallmarks of senior instructional 
design specialists. 

Reflective Techniques. Many individual and group reflection techniques were used during this workshop to 
help the content creators think about projects they were currently developing or evaluating. Some reflection began 
with relaxation techniques to get the audience to ‘let go’ of preconceived notions of what they were suppose to be 
doing and focus their thinking on what they know about their learners and the learning environment. This was 
followed by a reflective drawing activity where the 
participants were asked to draw a representation of 
the factors they believed most important in impacting 
learners in their context. Then, they were asked to 
reflect on their pictures and write a short narrative of 
how they saw instructional design principles, learning 
principles, and technology capabilities reflected in 
their drawings. Guided reflection was then used to 
prompt them to think deeply about the relationships 
among instructional design, learning, and technology 
in sound ID practices and to honestly assess their 
own practices. A series of questions was presented to 
help them think through their activities during a 
typical instructional systems design processes. At 
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each stage they were asked to write an example of task they completed and how they felt about it, it they believed it 
added value to the ultimate outcomes, and if they were inspired or motivated at the outcome of the stage. In other 
words they were asked to assess whether they truly believed that the work they were doing was going to make a 
difference and could it be used to inform other practitioners. They were then asked to share their example and 
thinking with colleagues sitting near them. These types case-based reflections were designed to encourage 
collaborative problem solving and practice tips sharing that is found in most effective ID communities. See figure 2. 

A Story from Work in the United States . . . 

The problem: Graduate students new to instructional sciences lack understanding of the relationship 
between instruction, learning, and technology and how this relates to work practices in ID. 

Storyc This story is about groups of newcomers to the ID community. The first year graduate level ID 
students, often a diverse group of full-time and part-time students with different academic backgrounds, are required 
to take core introductory learning and instructional design courses. They often have incomplete ideas about what ID 
is all about, what ID practitioners do, and how they practice based on theoretical understandings of learning. 

Students are thus engaged in a variety of activities to learn about, and practice the work of, instructional design 
professionals. Often the projects are based in some types of technology and require that the final deliverable meet 
content and design guidelines as well as represent the students understanding of their growing role in the community. 

Reflective Techniques : Relaxation, drawing, and writing reflection activities were used throughout the 
semester in several of the core ID courses. These techniques are used to help students develop reflective practices as 
they learn about the instructional sciences. Given this is a fast paced world that provides what seems like continuous 
multiple sound bites of information and activity allows very little time to think in holistic terms and reflect on what 
we do and why we do it. Students in these courses are engaged weekly in short sessions that prompt them to develop 
habits and routines helping them clear chaotic thoughts of the day and think in substantial, genuine, and authentic 
ways about the practices of their profession and the meaning it brings to them. For example, at the beginning of the 
semester of a core instructional deign practices course, students were encouraged to draw pictures of their 
conception of their definitions of instructional design (e.g., their roles, tasks, etc.) and then a picture of what they 
thought their clients’ or bosses’ definitions and expectations were of them as instructional designers. They we asked 
to refrain from using words in their drawings. Throughout the semester the students were asked to review their 
pictures often, sometimes making changes and sometimes adding short narratives on their thoughts and changes. 
Each reflection session was started with a short relaxation technique and reflections on what inspired or motivated 
them about the current class session. At times the students we provided with opportunities to share their thoughts 
before the class was dismissed. This longer process (throughout the semester) promoted students to consider this 
field as they were learning and practicing inside and outside of class. They were also encouraged to keep a formal 
written journal of their thoughts on the instructional sciences, what they were learning, and how their learning was 
inspiring their thinking and practices. Often students were encouraged to share their reflection during class 
conversations on specific topics. Such activities can move simple journaling and drawing activities from an 
individual assignment into a dialogue that can inform both the individual on their thinking and the entire class on the 
diversity of thoughts, ideas, questions, and meaning of ID practices. 

Conclusions 

“Reflection [and mediation in the classroom] increases our awareness of ourselves and 
the world around us... we reflect on and talk about our ideas... and stand at the 
boundaries of our world as we search for truth and influence practice... (p. 67) the fruit 
of reflection is meant to be shared.” (Litchmann, 2005, p. 81) 

Being a member of a professional community comes with expectations of full participation and good 
practice. All members are responsible for developing an understanding of, and participating in, the work of the 
community, and ultimately transforming and improving its overall practices. Part of good practice is being self- 
motivated to participate in quality ways by finding meaning in our work. Reflection is one tool, or perhaps one of 
our technologies , that helps us develop strong ID competencies and practices, a sense of meaningfulness and 
personal worth of our own work, and helps us leverage our knowledge of practice to others on our collective journey 
to frilly understand and address global issues of enhancing learning environments and providing opportunities for 
everyone to become productive members of our worldwide society. Good reflective practice takes time and 
commitment. It is integrated into our enculturation practices (education) and our work. It is individual and 
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collaborative with the best interest of those in the community and those who are impacted by the community in mind. 
These guidelines and examples are just that, ideas and reports of reflection in action in a single point observation. 

The results of the impact of these exercises is yet to be assessed. Nonetheless, reporting of such practices is meant to 
inspire others in our community to investigate these types of activities, explore on personal and professional levels 
these techniques as possible ways to enhance our community of practice. 
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Abstract 

The corporate e-learning in Korea has grown rapidly over the previous six years (2000-2005). This study 
argues that the main cause of this heightened interest in corporate e-learning in Korea was not that companies 
needed to provide high-quality training programs through the Internet but that the government took initiative to 
transform the state into an information-based society. The policies for quantitative growth with minimum levels of 
quality and uniformity have been dominant and have resulted in the lack of diverse e-learning types for authentic 
practices in workplaces. This paper suggests that in order to cope with the new competency requirements of 
employees, corporate e-learning should be guided both by governmental support and by company initiative. 

Keywords: Corporate e-Learning, Korea 



I. Introduction 

Open and distance learning in Korea had not been fully implemented and discussed prior to 2000. Until 
then, since 1972, systematic formal education for adult learners who did not have the opportunities to enter more 
traditional colleges had only been provided at such a mega university as Korean National Open University (KNOU). 
Traditional, face-to-face education had been the dominant mode of teaching, while distance learning made up only a 
small portion of the Korean educational system. However, nine cyber universities, which were established in 2001, 
and the new Internet correspondence training policy for corporate e-learning by the Ministry of Labor in 1999 
initiated profound changes in the Korean educational system (Lim, 2003). Now many Korean adult learners can 
pursue an education through the various learning technologies of distance learning. Online college courses are 
delivered via printed material, radio, television, MP3, PMP (portable multimedia player), and the Internet. Also, at 
present, distance corporate training programs are delivered by mail and online as e-learning. 

The unprecedented growth of corporate e-learning in Korea has been a major feature of distance learning 
since 2000. The Ministry of Labor reported that the growth rate of Internet correspondence training participants was 
6,281% over the past six years (19,653 in 1999 and 1,254,066 in 2005) (Ministry of Commerce et ah, 2006). While 
this astronomical figure can be explained in many ways (Lee, 2006) with positive results, its negative effects on 
corporate e-leaming and distance learning have also been noted. 

This study intends to discuss the causes and effects of the rapid growth, as well as explore directions for 
future research and practice with regard to corporate e-learning in Korea. To determine the current issues, the study 
focuses on the significant developments implemented by a government-funded research center, the Korean Research 
Institute for Vocational Education & Training (KRIVET), over the past six years (2000-2005). KRIVET impact on 
the development of corporate e-learning can hardly be overstated (Lee, 2006). The causes of those developments and 
various resulting problems are examined in this paper. Finally, based upon this analysis, future directions are 
suggested. 



II. Current developments in corporate e-learning in Korea: 2000 - 2005 

Corporate e-learning in Korea developed in earnest following the implementation of a new government 
policy in 1999 on Internet correspondence training. Since then, the development of the field can be measured in five 
ways: rapid quantitative growth, government initiatives, dominance of the tutorial mode, quality assurance, and 
high adoption rate among large corporations. These will be discussed in turn. 
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Rapid quantitative growth 

One of the salient features of corporate e-learning in Korea over the past years has been its rapid 
quantitative growth. In Table 1, it may be observed that the number of employees who participated in the Internet 
correspondence training program or e-learning greatly increased from 19,653 in 1999 to 1,254,066 in 2005 
(approximately 63.8 times more). In 1999, only 2.5% of the total number of trainees participated in the Internet 
correspondence training. But in 2005, that ratio increased up to nearly half of the total number of trainees (46.4%) in 
2005. 



<Table 1> Number of Trainees Supported by Employment Insurance Act(1999 -2005) 



Year 


Internet-Based 


Postal-Based 


Classroom-Based 


Total 




Internet 


Trainees (A) 


Increase 

rate 


Trainees 


Increase 

rate 


Trainees 


Increase 

rate 


Trainees 

(B) 


Increase 

rate 


^Training Rate 
(A/B*100) 


1999 


19,653 


- 


85,055 


- 


676,700 


- 


781,408 


- 


2.5% 


2000 


137,712 


600.7% 


161,825 


90.3% 


920,797 


36.1% 


1,220,334 


56.2 


11.3% 


2001 


406,159 


194.9% 


199,242 


23.1% 


950,001 


3.2% 


1,555,402 


27.5 


26.1% 


2002 


543,320 


33.8% 


197,045 


-1.1% 


843,958 


-11.2% 


1,584,823 


1.9 


34.3% 


2003 


629,930 


15.9% 


193,570 


-1.8% 


838,478 


0.7% 


1,661,978 


4.9 


37.9% 


2004 


929,771 


47.6% 


283,338 


46.4% 


790,354 


-5.7% 


2,003,463 


20.6 


46.4% 


2005 * 


1,254,066 


34.9% 


339,645 


19.9% 


1,171,630 


48.2% 


2,705,341 


35.0 


46.4% 



Source: Ministry of Commerce et al. (2006). E-learning White Paper 2005-2006. 

* The number of trainees in 2005 is projected on the data from the Ministry of Labor in Oct. 2005. 



The high growth was due to the expansion of the Employment Insurance Act, which earmarked financial 
support for e-learning programs. The law allowed the Ministry of Labor to begin in 1999 providing institutional 
support. Thus, the number of corporations and workers that participated in e-learning increased rapidly over the 
previous 5-6 years. Moreover, corporations took advantage of e-learning, giving more employees access to 
educational opportunities at relatively low cost. The large corporations quickly adopted the e-learning systems and 
invested money to develop programs. 

The rapid growth of corporate e-learning in Korea can be also attributed to an increase in theoretical studies 
on corporate e-learning. One of the leading journals on corporate training in Korea is the Journal of Corporate 
Education. This academic journal published its first volume in 1998 in the area of training methods, focusing 
especially on the applications of different technologies and programs. Not surprisingly, the journal has discussed e- 
learning with regular frequency since 1999. Of the 13 volumes of the journal, 26 of 79 articles (33%) have dealt 
directly with the subject of corporate e-learning. Considering the plethora of educational methods and issues in 
corporate training that could be discussed, devoting over 30% of the journal articles to e-learning is doubtlessly a 
testament to the rapid quantitative growth of the field as well as the high level of theoretical interest it has garnered 
in Korea. 

Government initiatives on corporate e-learning 

The Korean government, especially the Ministry of Labor, has played a significant role in developing the 
field of corporate e-learning. In accordance with the government’s strategic plan hatched in the mid-1990s to 
transform the country into a knowledge-based information society, the Ministry of Labor has been the primary 
driving force behind the implementation of e-learning for corporate training since 1998. That year the Ministry of 
Labor initiated a pilot project that tested e-learning based training courses, a project which led to the conclusion that 
correspondence training should include both Internet correspondence training and postal correspondence training 
and resulted in expansions the following year. This decision caused both corporate e-learning and distance learning 
for adults to grow rapidly in Korea in 2000 (Table 1). Furthermore, as alluded to previously, the Ministry of Labor 
established a special division, the e-Learning Center at KRIVET, to monitor the quality standards of the e-learning 
institutes and of the Internet correspondence training program as well as to establish a new support system and make 
recommendations. 

In addition to these early initiatives of the Ministry of Labor, subsequent measures and policies have also 
been effected since 2000 to boost corporate e-learning (Lee, 2006). In order to support corporate e-learning on a 
long-term basis, the Ministry of Labor and the e-Learning Center at KRIVET developed the Corporate E-Learning 
Mid-Period Development Plan, 2004 -2008, which proposed strategies covering various aspects of e-learning such 
as servicing and maintenance of the system, infrastructure construction, cultivation of human resources, 
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standardization, and quality control. Recently, Internet correspondence training regulations have also been revised to 
incorporate ‘blended learning’ and a scaled training fee structure, based on an analysis of the quality of the e- 
learning institutes. Lastly, the worker’s tuition support system has been modified as well so that individual workers 
can receive financial support when they register independently for e-learning programs. 

Dominance of tutorial e-learning 

Most corporate e-learning courses in Korea are in tutorial format in which major points are supplemented 
with elaborations and examples (‘Intro’ and ‘Lesson’ in Figure 1) followed by practice problems (‘Activity’ in 
Figure 1). Various design strategies have been applied — for example, Keller’s ARCS (Attention, Relevance, 
Confidence, Satisfaction) model has been adopted to enhance effectiveness and for animated presentations advanced 
technologies such as Flash have become standard authoring tools. 
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[Figure 1] An example of e-Learning: Tutorial type 



Table 2 shows that nearly 90% of all e-learning content in 2005 were tutorials in either HTML 1 or Lecture- 
on-Demand (LOD) format. The remaining 10% were simulations that honed technical skills. As these two types — 
tutorials and simulations — became so ubiquitous over the past three years in the field of corporate e-learning, the 
government’s Employment Insurance Fund eventually dropped support for other types of e-learning programs. 



<Table 2> e-Learning Program Proportion by Types from 2003 to 2005 



Types 


2003 


2004 


2005 


Self developing 


Outsourcing 


Self developing 


Outsourcing 


Self developing 


Outsourcing 


HTML 

Tutorial 


545 (88.0) 


1,086 (79.3) 


738 (88.0) 


1,005 (68.6) 


742 (79.7) 


1,082 (80.8) 


LOD 


26(4.2) 


212(15.5) 


33(3.9) 


315 (21.5) 


83 ( 8.9) 


138(10.3) 


Simulation 


48 ( 7.8) 


72(5.3) 


68(8.1) 


146 (10.0) 


106(11.4) 


119(8.9) 


Total 


619 


1,370 


839 


1,466 


931 


1,339 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 


(100.0%) 



Source: Lee, S. et al. (2006). Evaluation of Internet Correspondence Training Institutes, Year 2005. Seoul: KRIVET 



Quality assurance 

The government, the Ministry of Labor in particular, has tried to assure e-learning quality over the previous 
few years. In 2002, the e-Learning Center at KRIVET launched an assessment system to judge the appropriateness 

1 HTML in this Table refers to Web-based instruction, in which text, graphics, and some animated objects are 
displayed in HTML format. LOD(Lecture-on-Demand) refers a type of e-Learning in which lectures by an instructor 
are recorded and delivered through the Internet at learner’s convenience. 
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of Internet correspondence training programs. All e-learning programs supported by the Employment Insurance 
Fund were required to be evaluated by the Center. As seen in Figure 2, the B level grew annually from 16.5% in 
2002 to 48.1% in 2005, demonstrating that instructional design and content quality were indeed improved. 
Conversely, the proportion of D and F level steadily decreased from 27.5% in 2002 to 5% in 2005. The quality of the 
e-learning programs was successfully improved by implementing the assessment system. 

(unit: %) 




[Figure 2] e-Learning Program Assessment Results( 2002 - 2005) 
Source: Ministry of Commerce et al. (2006). E-Learning White Paper 2005- 2006. 



High adoption rate in large companies 

Participation in corporate e-learning programs has not been evenly distributed across the corporate sector. 
The employees of larger corporations comprise the largest proportion of Internet correspondence training system 
participants. Table 3 shows that in 2004 just 8% of training participants from the assembly line in small and medium 
companies participated in Internet correspondence training programs, while nearly 30% of those from large 
companies did so. 



<Table 3> The Proportion of Training Types for Assembly Line Workers by Company Size 



Company 



Classroom 

Training 



Internet 

Training 



Postal 

Training 



Small & Medium 
Company 
Consortium 



(Unit: number, person) 
Field training Total 





Programs Trainees 


Programs Trainees 


Programs Trainees 


Programs Trainees 


Programs Trainees 


Programs Trainees 


Small and 


1,847 


14,432 


432 


1,877 


300 


6,386 


3 


22 


8 


90 


2,590 


22,807 


Medium 


(71.3%) 


(63.27%) 


(16.8%) 


(8.2%) 


(11.6%) 


(28%) 


(0.1%) 


(0.1%) 


(0.3%) 


(0.4%) 






Large 


4,561 


68,565 


1,145 


34,294 


865 


14,401 


1 


1 


4 


6 


6,576 


117,267 


(69.4%) 


(58.5%) 


(17.4%) 


(29.2%) 


(13.2%) 


(12.3%) 


(0.02%) 


(0%) 


(0.1%) 


(0%) 







Source : Ministry of Labor, Unemployment insurance DB, 2004. 



Table 4 further shows that the implementation ratio decreases significantly in relation to smaller company 
size. In small companies (less than 50 employees), only 10.3% of workers participated, whereas 52.4% of 
employees did so in large companies (140-299 employees) (Jang & Yoo, 2006). 



<Table 4> e-Learning Program Proportion Change by Company size in 2006 







Company size 




total 


difference 


Category 


Less than 
50 Workers 


50 Workers to Less 
than 149 Workers 


150 Workers to Less 
than 299 Workers 


e-Leaming Yes 


8 ( 10.3) 


15(24.6) 


22 ( 52.4) 


45 ( 24.9) 




Program No 


70 ( 89.7) 


46 ( 75.4) 


20(47.6) 


136( 75.10 


X =23.938 


total 


78(100.0) 


61 (100.0) 


42(100.0) 


181 (100.0) 


p<.01 



Source: Jang, M. H., & Yoo, S. J. (2006). E-Learning Needs Analysis for the Employees of Medium and Small Companies. 
Journal of Vocational Education & framing, 9(3), 175-202. 
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The Internet correspondence training system appears at first glance to provide broader access to educational 
opportunities in Korea. It seems to put training within reach of those employees who would not otherwise have such 
an opportunity. However, in realty, it fails to achieve that goal. Assembly line workers and those employed by small- 
or medium-sized companies have not received equal opportunities for the Internet correspondence training. 

III. Issues in corporate e-learning in Korea 

Although there have been great developments in corporate e-learning in Korea over the past six years, there 
remain certain key problems in the field: moderate quality’, uniformity and controlled growth, limited evaluation 
criteria, and unevenly distributed adoption rates. 

Moderate quality of corporate e-learning 

The quality of corporate e-learning has become an issue that has only been compounded by the rapid 
growth rate. As mentioned, since the 1999 introduction of e-learning in Korea two formats have emerged as 
preeminent: one is LOD(Lecture-on-demand) in which the presentation of a lecturer is recorded in motion picture, 
and the other is Web-based instruction in which text, graphic, and some animated objects are displayed on Web 
pages as the learners click to progress. LOD has become a dominant format because not only it was easy and 
relatively cheap to develop but it also conformed to expectations of what training should be (i.e. lecture delivery’ at 
learners ’ convenience). A small percentage of e-learning was developed as web-based instruction and most of this 
type were just copies of printed materials without learners’ active participation. These problems were partially 
remedied when the new assessment system was introduced in 2001. As Table 5 illustrates, growth stalled in 2002 
(the number of total institutes decreased from 1 10 to 93) with the introduction of governmental regulations, of which 
the assessment system was a key provision. The system succeeded in upgrading the quality of tutorial-type e- 
learning programs at the expense of other formats. The quality issue will be discussed in more detail in the latter 
portion of the quality assurance section. 



<Table 5> Number and Increasing rate of Corporate e-Learning Institutes in Korea( 2001 - 2005) 



Year 


Self-developing 


Outsourcing 




Total 


year 


Institutes 


Increasing 

rate 


Institutes 


Increasing 

rate 


Institutes 


Increasing 

rate 


2001 


56 


- 


54 


- 


110 


- 


2002 


57 


1.8% 


36 


-50.0% 


93 


-18.3% 


2003 


55 


-3.6% 


33 


-9.1% 


88 


-5.7% 


2004 


53 


-3.8% 


55 


40.0% 


108 


18.5% 


2005 


68 


22.1% 


80 


31.3% 


148 


37.0% 



Source: Ministry of Commerce et al. (2006). E-learning White Paper 2005-2006. 

Notes: 1 . If a company provides both intra-company and outsourcing programs, it will be estimated as outsourcing training. 



2. The number of companies conducting more than one training per year. 

Uniformity and controlled growth 

The government initiative on corporate e-learning in Korea has yielded unexpected results. Although it 
stimulated the quantitative increase of corporate e-learning programs in a short period of time, one mode of e- 
learning, tutorial, dominated the e-learning landscape. While the Evaluation System for Contents and Design Quality’ 
of e-Learningby the e-Learning Center ( Table 7) succeeded in upgrading the basic quality of the tutorials, it failed 
to encourage the development of diverse modes of e-learning programs beyond simple tutorials, and also neglected 
to support new ideas and studies to help e-learners become self-regulated or self-directed(Lim, 2005). Many e- 
learners did not successfully complete the e-learning programs in which they had enrolled; they dropped off early on 
or midway through because they were not self-motivated. 

Although tutorial is an effective instructional type for certain objectives, it cannot support some essential 
objectives and activities, including teaching problem-solving skills, creative thinking skills, and self-directed 
learning. Yet these are the very skills and experience expected of workers in Korea’s knowledge-based society. 
However, corporate e-learning in Korea currently has not successfully promoted different e-learning models such as 
Problem-Based Learning, Goal-Based Scenario, and Case-Based Learning (Kang et al., 2006). 

In addition, the government, by virtue of their initiative, regulated the growth of e-learning in a way that 
stifled spontaneous innovation. Most companies assumed a passive role in designing and developing other types of 
e-learning. The rapid growth of e-learning was achieved without autonomous efforts from the corporate sector. For 
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instance, the possibility of integrating e-learning with the knowledge management system in a company (Rosenberg, 
2001) or with long-term blended learning strategies has not been systematically examined, in spite of the high 
demand. 

On the other hand, some alternative designs have been explored from a theoretical standpoint. The Journal 
of Corporate Education in Korea has treated many topics related to course design, and secondly, to learner support 
and cost issues when those articles from the journal were analyzed by an analytical framework of Rha and Han 
(2002) 2 (Table 6). As this journal mainly focuses on the educational methods in corporate training, especially from 
the perspective of educational technology’, this is not an unexpected result. The course design topics covered in the 
articles included ‘strategies for learning motivation in e-learning,’ ‘design strategies for Goal-Based Scenarios,’ 
‘blended learning strategies’, and ‘A design model for e-learning.’ On the other hand, as e-learning is relatively new 
to adult workers, learner support issues have been widely discussed. The issue of cost has also been examined 
because the corporate sector has been particularly interested in determining whether e-learning is cost-effective. 

<Table 6> The Topics of e-Learning Studies in Korea (1998 -2006) 



Topics Frequency Ratio 



Course Design 


11 


42.3% 


Learner support 


6 


23.1% 


Cost 


4 


15.4% 


Learner participation 


3 


11.5% 


Learning Contents 


1 


3.8% 


Other 


1 


3.8% 


Total 


26 


100.0% 



Note: The articles from the Journal of Corporate Education in Korea were selected for examination 



In addition to the Journal of Corporate Education other academic journals and periodicals have also 
discussed topics related to e-learning design. The Journal of Educational Technology’ in Korea, for example, has 
dealt with issues of corporate e-learning from the perspective of educational technology, notably, issues such as 
tutoring (Cho & Lee, 2004) and supporting self-regulation (Lim, 2005) have been areas of recent focus. 

Limited evaluation criteria 

It can be argued that the current perception of quality of corporate e-learning can be attributed to the narrow 
and ambiguous evaluation criteria. Table 7 shows the current evaluation criteria for e-learning programs. It consists 
of five dimensions: instructional design, interaction, evaluation, instructional support design, and technology’. 
Whereas the dimensions and the sub-criteria do cover important quality aspects of e-learning programs, some 
problems still exist. First, they focus too narrowly on evaluating the tutorial type of e-learning, virtually assuming 
that the tutorial format is the only one to assess; other types of e-learning programs such as Case-Based Learning or 
simulation have not been easily evaluated under these criteria. Second, they do not provide specific guidelines for 
each criterion. The criteria were ambiguous and judgment could vary depending on the evaluator. More specific 
guidelines in a rubric format should be developed to make the evaluation more objective and effective for future e- 
learning programs. 

<Table 7> The Evaluation Criteria for e-Learning Programs 

Dimensions Criteria 

Instruction Design Content presentation strategy and method relevancy 

Suitability for supplementary & advanced learning resources 
Interaction Appropriateness of learner-instructor interaction 

Appropriateness of learner-learner interaction 
Appropriateness of learner- contents interaction 



The framework suggests course design, learner support, cost, learner participation, and learning contents as the major 
components of e-Leaming. Course design includes ‘the systematic use of instructional strategies’, ‘the flexible design and change 
of course’, ‘active interaction’. Learner support deals with ‘easy admission and graduation’, ‘counseling system’, and ‘stable 
infra structure and technology’. Cost involves such areas as Tow tuition fee’ and ‘cost-effectiveness’. Learner participation 
examines ‘learner involvement’, ‘learner satisfaction’, and ‘free choice of course’. Finally, learning contents deals with 
‘interesting content’, ‘easiness to accessing information’, and ‘usefulness of content’. 
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Appropriateness of learner-program manager interaction 
Evaluation Appropriateness of method 

Instructional Support Design Appropriateness of help function 

Connectivity of navigation (Interface) 

Technology Technical stability 

Source: e-Leaming Center in KRIVET (2006) 

Unevenly distributed adoption rates 

As noted previously, the implementation of corporate e-learning in Korea has mostly been confined to 
larger companies. Employees of small- and medium-sized companies have not yet fully experienced e-learning 
programs. To overcome this shortfall, the Ministry of Labor’s worker’s tuition support system has been recently 
revised to provide financial support for these underserved workers (Lee, 2006). Further consideration in terms of 
more access to Internet correspondence training system is needed in order to improve corporate e-learning in Korea. 

From a theoretical vantage point, selected studies in the Journal of Vocational Education & Training by 
KRFVET have discussed the issue of supporting medium and small companies, such as by beginning with an e- 
learning needs analysis of these companies (Jang & Yoo, 2006) and an assessment of e-learning course selection 
criteria (Kwon, Lee, Rha, & Lim., 2006). These studies were mainly conducted to come up with political 
implications for the future of e-learning implementations at the national level. 

IV. Conclusions: Future directions and prospects 

The current development of corporate e-learning in Korea was examined in terms of five aspects: Rapid 
quantitative growth, government initiative, dominance of the tutorial mode, quality assurance, and high adoption 
rate among large corporations. Each one of these also has a corresponding weakness: Rapid quantitative growth has 
meant that quality has at best been moderate. Government initiative has fueled complacency as companies have 
hesitated to develop their own e-learning programs for specific purposes such as problem-solving or creative 
thinking skills. The dominance of the tutorial mode has kept other types of e-learning from being actively designed 
and implemented. Quality assurance measures have used evaluation criteria so narrow and ambiguous that other 
types of e-learning contents could not be easily evaluated and the evaluation was susceptible to subjective influence. 
And finally, the high adoption rate among large corporations has translated into workers of small and medium 
companies not having equal opportunities. 

Nevertheless, corporate e-learning does seem to have had a great impact on the recent development of life- 
long education and distance education in Korea. While the ratio of participants in life-long education in Korea has 
been relatively low among OECD countries (OECD, 2000), corporate e-learning has played an important role in 
increasing the rate of participation rate in a short period of time. This increase was mainly due to the government 
initiative to transform the state into an information-based society where all the aspects of government, including 
education and training systems, rely on information infrastructure and environments. The Ministry of Labor was no 
exception, and it took advantage of the Employment Insurance Fund to achieve its goal for contributing to the 
information-based society. Companies were permitted to ask for government subsidy as long as they provided e- 
learning programs for their employees. Implemented in 2000, this allowance had a significant effect on the growth 
of corporate e-learning. 

In this respect, corporate e-learning in Korea has a unique distinctive. The main impetus for its rapid 
growth has not been that companies needed to provide high-quality training programs to more workers but that the 
government took initiative to transform the nation into an information-based society. Most e-learning programs were 
tutorials that could be made easily, and their common objectives were for acquiring knowledge or understanding 
content(Lee et. al., 2006). Companies wanted to ensure that employees could do their work competitively or show 
competency in new areas. They did not want any more workers who just understand basic, factual knowledge. 
However, competency-related e-learning or performance-based e-learning has not been fully examined yet. 

Corporate e-learning can be improved in two ways. First, it should be dynamic rather than static, serving 
the new requirements of companies and employees. If it remains confined to traditional modes of education or 
schooling where understanding knowledge can be acceptable as an educational goal, it will be criticized and 
eventually phased out. It should be responsive to demands for new skills, competencies, or performance, training 
objectives that can improve employee effectiveness and efficiency. Therefore, the current government e-learning 
initiative should be changed into one that is directed by both government and companies, one in which companies 
play an active, executive role and have a vested interest. Each company should be encouraged to develop and 
implement advanced and authentic programs autonomously, programs such as an e-learning version of Problem- 
Based Learning or Case-Based Learning (Kang & Oh, 2006). And the evaluation criteria should become more 
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inclusive and go beyond examining the components of the tutorial. It should stimulate new trials and developments 
of corporate e-learning programs. 

Second, corporate e-learning is a kind of distance learning. Theoretically this provides more possibility to 
help adult learners access education and training opportunities. Corporate e-learning can provide training to 
employees who might otherwise never have had the opportunity to train because of their time and space constraints. 
The initial implementation of corporate e-learning was geared toward large corporations such as Samsung, LG, and 
SK. These were relatively eager to adopt the program because they considered e-learning to be cost-effective and to 
provide more training for their employees(Jang & Yoo, 2006). Yet small- and mid-sized companies were not able to 
offer these programs due to financial constraints. Moreover, they did not recognize the value of e-learning for their 
workers. To solve this inequality problem, the government should play a key role in encouraging the implementation 
of e-learning. The Bureau of Small- and Mid-Sized Companies in the Ministry of Labor must come up with effective 
strategies to mitigate the imbalance, while the consortium of affected companies should take ownership of 
implementing e-learning. And, government and companies should collaborate to provide leadership aimed at 
providing opportunities for the disadvantaged employees of these companies. 

As mentioned, the unequal access to e-leaming should be acknowledged and countermeasures to provide 
more equal opportunities should be devised, especially for employees for whom e-Learning is relatively new 
(Piskurich, 2003; Lim, 2005). Considering their schooling experience that focused on passive learning, it is easier to 
see how it may be difficult for such employees to manage their independent learning, or exercise appropriate self- 
direction or self-regulation(Lim, 2005) in learning. Providing facilitator intervention or even a new learning 
management system to encourage and support learner’s self-regulation systematically are steps that could mitigate 
the problems. This issue in terms of supporting learner’s self-direction or self-regulation should be studied and 
examined in further research. 
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The development of a method of collecting digital portfolios for Chinese 
learners of Japanese on an e-learning system 

Chunchen LIN 

Tokyo University of Foreign Studies 

1 . Introduction 



As the use of computers and the Internet has spread widely, collecting learner portfolios, especially digital 
ones, quickly received much attention: the main reason for this is that they are easy to collect with information 
technology. In the area of language teaching, many researches have been done within the framework of conventional 
language education and second language acquisition studies. The conventional ways of evaluation most often use a 
set of standardized tests to measure learners’ achievement levels. In contrast, collecting and analyzing digital learner 
portfolios enables us to evaluate how they have made progress, as well as to measure their achievement levels. In 
Europe, as European countries were integrated into the EU and more and more people were expected to move within 
the EU, a new method of measuring linguistic proficiency, called the Common European Framework (Trim et al. 
2002), was developed. In this framework, linguistic proficiency is measured by the European Language Passport. 
By using portfolios, language teachers can collect and store learners’ performance data directly from the classroom for 
a long period of time and in a systematic manner, which enables them to apply the data to developing language 
pedagogy and to establishing more advanced methods of measuring linguistic proficiency. 

Based on these assumptions, we maintain that it is effective to collect digital learner portfolios on a new 
task-based e-learning system. In this system learners are given systematically organized tasks through the four years at 
university, their answers are collected via the system, and they complete their Language Passport with the help of their 
teachers. Teachers can, on the other hand, analyze the long-term data of learners so that they can apply their analysis to 
better methods of measuring linguistic proficiency. This system focuses on the characteristic features in language 
education: it is designed to collect the data of how learners understand the meaning of speech sounds and the written 
characters, how their utterances are given based on the input, and how they write down what they have listened to. 
Teachers just need to specify the task type, create the questions, and post them on the e-learning system, in order to 
collect data through the system which automatically gives tasks for learners to do and make answers to. 

This paper describes the system we have developed, and the data that can be collected through the system. 



< 

sounds 



Input to learners 



2. Design of the system 

* K Sounds and videos 



Type 2 



Type 1 



Type 4 






Type 3 


Type 6 






Type 5 



Output of learners 



characters 



Characters and pictures 

Figure V Six tasks 
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Language learning consists of the four skills: listening, speaking, reading, and writing. Listening and 
reading are input to the learners, while speaking and writing are output from them. So this system provides the input to 
the learners, and then collects the output that the learners give to the input. The design of this system is given below. 

2.1 Task types and XML-formatted data 

If we divide the four basic skills in language learning to input (listening and reading) and output (speaking 
and writing) as we have just mentioned, each task type will consist of a combination of input and output as shown in 
Figure 1. This system gives tasks in which questions are put using characters, sounds, pictures, or videos, and the 
learners’ answers are collected in the form of characters or sounds. We have six task types in the current system as 
shown in Figure 1; we have excluded two types that offer video materials to the learners in order to avoid overload of 
the system. 



type 


Offered data 


Collected data 


Data given to learners 


Learners’ output collected 
through: 


Example of the task 


Type 1 


sounds 


characters 


directions and sounds 


text box 


dictation 


Type 2 


sounds 


sounds 


directions and sounds 


voice recording tool 


oral quiz 


Type 3 


characters 


characters 


Directions 


text box 


reading quiz 


Type 4 


characters 


sounds 


Directions 


voice recording tool 


reading out loud 


Type 5 


pictures 


characters 


directions pictures 


text box 


speaking what one thought 
about the pictures 


Type 6 


pictures 


sounds 


directions pictures 


voice recording tool 


speaking what one thought 
about the pictures 



Table 1 : Description of each task type 



As shown in Table 1, when we create tasks, we do not have to make so many questions. Also, once we 
decide which task type to use, the interface for learners has only to have text boxes and a voice recording tool. When 
we want to put questions in Type 1 (dictation), for example, we just need to give directions and sound files; Type 6 
(speaking what one thought about the pictures) requires us to just give directions and the picture files. The data 
structure of the learners’ output is decided upon according to the task type. We defined the data structure in the XML 
format. Table 2 shows samples of the XML-formatted data for Type 1 and Type 6. 



<?xml version-' 1.0" encoding="UTF-8"?> 


<?xml version-' 1 .0" encoding="UTF-8"?> 


<portfolioTl xml:lang="ja"> 


<portfolioT6 xml:lang="ja"> 


<name>dictation of a conversation on the 


<name>oral descriptions of pictures</name> 


Intemet</ name> 


<question> 


<question> 


<explain>See the pictures below and explain the people in the 


<explain mainlangl="none">This is a conversation on the 


pictures in Japanese. Record your speaking with the voice 


Internet. Press “play” and write it down. 


recording tool below, and write it down in the box.</explain> 


</explain> 


<ques answrite— '1" anstime="30"> 


<ques mainlang2="none" anslang— 'none"> 


<examPic>encoded picture data l</examPic> 


<quesExp></quesExp> 


<examPic>encoded picture data 2 </examPic> 


<examSound>encoded sound data 


</ques> 


</examSound> 


</question> 


</ques> 


</portfolioT6> 


</question> 


(Type 6) 


</portfolioTl> 




(Type 1) 





Table 2: XML-formatted data samples for Type 1 and Type 6 



2.2 Data entry 

Multimedia contents such as sounds and pictures are encoded and inserted into the XML-formatted data as 
binary data for convenience. We use Microsoft InfoPath to enter the data into the system. We can enter character data 
into the form, and select files for multimedia contents such as sounds and pictures, which are automatically converted 
to binary data and inserted into XML-formatted files. Figure 2 is the screenshot of InfoPath when we enter data in 
Type 1. 
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2.3 Operation of the system on an e-learning environment 

When we collect digital portfolios to create Language Passport for learners, it is efficient to use an e-learning 
system that can distinguish learners. We have developed a system on which we can post the XML data described above 
on the TUFS e-learning system that was developed in our university. Figure 3 shows the screenshots of the materials 
posted on the system. Figure 3(a) shows a Type 6 task, in which learners see the picture, record what they thought 
about it with the voice recording tool, and submit the data, which will be automatically collected. This example is a 
task where Chinese learners of Japanese see a picture of a 
family and they are required to explain the relationship 
among the family members in Japanese. Tasks in Types 2 
and 6 have a text box at the bottom of the page where the 
learners can write down what they speak, which can render 
this data easy to analyze afterward. Figure 3(b) shows a 
Type 1 (dictation) task. Learners press the “play” button, 
listen to the questions, and write their answers into the text 
box at the bottom of the page. This example is a task where 
learners are required to write down a conversation between 
a Chinese learner of Japanese and a native Japanese 
speaker. 
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Figure 3(a) Figure 3(b) 

Figure 3 : Materials posted on the system 

The data collected through the tasks are provided to the teachers and learners in the format shown in Figure 
4. Figure 4 shows the data collected in the tasks in Type 1 described above. Teachers and learners can obtain the data in 
the Microsoft Excel format, where the data is linked with the sounds and pictures given to learners. 
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Figure4 The Data collected in the tasks in Type 1 
3 . Summary and future work 

In language teaching, the Language Passport can be a more effective method of measuring linguistic 
proficiency than traditional paper-based tests. Collecting and analyzing task-based learner data covering the four basic 
skills can enable more accurate evaluation of the learners’ progress. This paper explained the task types and how we 
can collect portfolios through an e-learning system. Examples of the tasks given to Chinese learners of Japanese were 
also provided. We plan to keep collecting portfolios mainly at the levels of Al, A2, and B1 in speaking that are 
stipulated in the Common European Framework. Analyzing the digital portfolios for learners of Japanese collected in 
this study will enable us to discover new methods for language teaching. 
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Developing an Educational Game of Tangram with Tangible User Interface 
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Tangram, an inherited historical legacy, is widely used in learning shapes and stimulating 
creativity in many countries and areas for both of its shape simplicity and combinatorial complexity. Tangram is 
an excellent learning material to enlighten children’s intelligence and thinking. It helps children to establish the 
connection between physical objects and geometric forms as well as cultivating imagination, observation, shape 
analysis and logic. Teachers use tangrams to aid children to learn the general concept of form, vision difference, 
cognitive technique, vision memory, coordination of hands and eyes, divergent thinking and human relationship. 
Aside from finding solutions by arranging pieces into a predefined shape, tangram can be used to build creative 
shaped without any limitations. Though generally tangram is played by a single child, team work is another 
interesting way to play with such that children can learn how to cooperate and communicate to accomplish a 
task. 

Like many learning materials, digital tangrams had been appeared almost at the time when there 
were computer games. A multimedia tangram with animation and audio feedback brings more pleasure to the 
children that can not be found in traditional physical tangrams. However, there are drawbacks for digital tangram 
games. First, long time of sitting in front of a computer is unhealthy. Second, fixed types of video and audio 
stimulations are insufficient for the development of child’s senses. Third, traditional human-computer-interface 
device (mouse and keyboard) may not suitable for children who are unfamiliar with the techniques to operate 
these devices. Finally, traditional interface is suitable for a single player. It is hard to let multiple players to 
control separately in the same game. 

Tangible User Interface (TUI) is a way to expand the traditional human-computer- interface (HCI) using 
regular input/output devices such as mouse, keyboard and monitor to tangible (graspable) physical objects 
related to the content of the system. Unlike using an HCI that the user have to memorize the menu and buttons 
and the correspondence with mouse movement and keystrokes, the user of a TUI can focus on the meaningful 
objects and manipulate them intuitively. Information engineers have designed many creative applications with 
TUI to relief the restriction of traditional interface devices. The most natural and intuitive way for a child to play 
with a tangram is moving physical pieces with hands instead of moving virtual pieces with a mouse and a 
keyboard. 

Following this concept, we developed an educational game of tangram with tangible user interface. The 
shape of a tangram puzzle is projected on a table with a projector connected to a personal computer. Traditional 
physical tangram pieces on the table can be moved by the hands of the game player. A webcam is also connected 
to the computer to scan the pieces on the table. Detection is performed to test if the pieces match the projected 
shape. If the projected shape is matched in the specified time, animation and audio feedback are generated as 
reward. 

In the scenario described, physical world (tangram pieces and table) and virtual world (projected puzzle) 
are merged together into a more intuitive interface that even a child without any knowledge of mouse 
manipulation and keyboard input can still enjoy the fun of the game of tangram. The advantages of physical 
icons and digital content are retained and merged. The player has to move his/her arms and fingers to move the 
tangram pieces. Sometimes the player even has to stand up and walk around the table to observe the projected 
puzzle. Thus prevent the player from the unhealthy status of sitting statically for a long time. By moving pieces, 
the muscles controlling fingers, hands and arms are exercised. The sense of touch is stimulated by moving and 
touching pieces. Since the action of moving pieces is intuitive, there is no need to spend time to get used to the 
interface like the traditional interface. Most interestingly, the physical tangram pieces can be manipulated by 
several players. Therefore the restriction of a single player in a digital game of tangram is removed. Several 
players can play the game at the same time. Children can learn social cooperation and competition through the 
multi-player mode. 
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It is discovered that, with the introduction of tangible user interface, the advantages and features of 
physical objects and digital content can be combined together and transformed into an intuitive and creative 
interface for education and learning. 
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Abstract 

This study examined strategies that experienced instructional designers use to obtain stakeholder buy-in, which prior 
research has not examined. The results of this research showed that experienced instructional designers consider 
this to be of paramount importance. Results revealed that the most common strategies employed include using 
design documents, establishing checkpoint reviews, providing design options for stakeholders, using pilot studies, 
and following up with stakeholders to determine their level of satisfaction with the intervention. 

Introduction 

This research examined strategies that experienced instructional designers use to obtain stakeholder buy-in 
during the instructional design process. Instructional Design (ID), or Instructional Systems Design (ISD), concerns 
itself with the systematic series of steps that are used to create instruction. Several prescriptive models for how ID 
should be carried out have been proposed over the years, but the one most commonly cited by practitioners is the 
“ADDIE” model. This model consists of 5 phases: Analysis, Design, Development, Implementation, and Evaluation 
(ADDIE). An instructional designer is a person who engages in this process in order to create an instructional 
intervention that addresses specific learning needs on the part of the target audience. 

As with any prescriptive theory, however, actual practice may differ. Evaluating prescriptive theories in 
light of actual practice is therefore useful for advancing the development of new prescriptive theories. This research 
examined the actual practice of instructional designers in the business/industry setting to determine the specific 
strategies that they use to obtain stakeholder buy-in. Stakeholders are typically clients or sponsors, and the learners 
themselves. The need for this research is driven both by the researcher’s experience and that of other instructional 
designers with whom he has spoken, as well as support in the literature for the important role of stakeholders in the 
instructional design process. While existing research affirms the important role of the stakeholder in the 
instructional design process, very little explicitly addresses how experienced instructional designers go about 
obtaining stakeholder buy-in, the situations in which they do so, and how the context determines their strategies. 

This research benefits all who practice instructional design, and contributes to the body of research on effective 
instructional design. 

Several studies have examined the practice of instructional designers. Generally, these studies examine the 
practice of novice instructional designers, novice as opposed to experienced instructional designers, or experienced 
instructional designers. Among these studies, there are a combination of experimental (in which the designers are 
given an instructional design task), quantitative (surveys), and qualitative (asking the designers to reflect on previous 
instructional design experiences). These studies typically examine either how instructional designers work, or why 
(examining their decision-making process, or the learning theories they use). 

Kerr (1983) conducted an experiment with 26 novice instructional designers to examine the decision- 
making process at different points in the instructional design process. Rowland (1992) had a total of 8 novice and 
expert designers think aloud during a design activity in order to determine what happened within the design process 
from a problem assessment, solution generation, and decision-making perspective, and the differences between 
experts and novices. Among other conclusions, he found that experts differed from novices in that they made a 
lengthy analysis, used a variety of interventions for the solution to the problem, and based their decisions on 
multiple, global factors (as opposed to single, local factors). Perez and Emery (1995) also investigated differences 
in novice and expert thinking with 4 expert designers and 9 novice designers through interviews with experts and a 
think aloud with the novices, finding as Rowland that experts spend more time exploring the problem and consider a 
wide range of factors in their solution. LeMaistre (1998) examined differences between novice and expert thinking 
as well, with findings similar to the others. 

Of the studies examining instructional design practice, several are oriented towards determining those 
activities which are employed in professional practice. Wedman and Tessmer (1993) used a survey to gather data 
from 73 designers to determine the design activities they apply in their projects. The researchers found that 
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designers selectively follow ID model prescriptive guidelines, and that the activities of pilot testing and establishing 
a need for training were most often omitted (however, they did call for the increased use of pilot testing). Winer & 
Vasquez-Abad (1995), in a replication of the Wedman and Tessmer study, surveyed 66 designers to determine the 
degree of use of specific ID activities and the factors influencing their use. Among the results, they found less 
emphasis on conducting up-front in-depth analysis (such as a needs assessment) and more emphasis on an iterative 
process of testing and modification. Pieters and Bergman (1995) also used a survey to determine which activities are 
practiced by designers. Key among their results was the finding that designers consider it important to communicate 
with stakeholders in order to know how receptive they are to different solutions. 

Visscher-Voerman (1999) also studied the activities that instructional designers employ in practice and why 
they deviate from their general project approach. Visscher-Voerman conducted interviews with 24 expert designers 
from the Netherlands, and found 16 design principles. Four of these principles make specific reference to obtaining 
stakeholder buy-in (several others relate to early prototyping and the importance of formative evaluation, which 
might be important for obtaining stakeholder buy-in). These four principles are: 

1) During the design process, designers should pay as much attention to creating ownership with clients 
and stakeholders, as to reaching theoretical or internal quality of design 

2) Designers should not only ask clients and (future) users for content-related input, but should also 
give them the right to decide about the design itself 

3) Designers should ask those with an important role in the development and implementation for their 
early participation in the design activity 

4) A useful means to help clients, partners, and other stakeholders to choose a solution and to formulate 
product specifications is by showing products from former projects. 

The importance of the latter strategy as a means to obtaining stakeholder buy-in is supported by Kirschner, Carr, and 
van Merrienboer (2002), who used Visscher-Voerman’s sixteen principles in a survey of instructional designers. 

Several of the studies mentioned above assert the importance of a highly iterative, prototype-based design 
process. Such a practice might include stakeholders in the process, but this is not the explicit focus of these studies. 
Visscher-Voerman (1999) does provide support for the importance of obtaining stakeholder buy-in, noting several 
strategies related to obtaining it, and Pieters and Bergman (1995) and Kirschner et al. (2002) note the importance of 
communicating with stakeholders to assess their receptivity to potential solutions. Within existing research, 
however, there is not a specific assessment of the strategies that experienced instructional designers use to obtaining 
stakeholder buy-in. Research in this area would provide a more in-depth perspective on what previous studies assert 
to be an important part of the instructional design process. 

Methods 

The researcher interviewed five experienced instructional designers who have practiced in government, 
business, non-profit, or education settings. Each interview lasted approximately 1 14 hours. Each subject had an 
average of 16.6 years of relevant instructional design experience; the minimum was 10 years and the maximum was 
20 years. Those interviewed had performed ISD both from within organizations (as an employee) and from outside 
organizations (as paid, contract-based, external consultants), though participants spoke for the most part of 
experiences when they were external to the organization. The participants were volunteers. 

Interview questions followed an interview guide approach (a guideline of topics to be covered, rather than a 
specific list of questions), which allowed for latitude in questioning, and exploration where needed. The participants 
were informed at the outset that their names would not be used in any published research. All participants were 
asked not to reveal the real names of the organizations referenced in their interviews, instead saying “one company” 
or “Company A,” etc. Following this, and an introduction to the nature of the research, the researcher asked the 
following questions: 

1. Describe your most recent instructional design project. What were the specific steps that you 
followed? 

2. Who where the stakeholders in the project? What were their initial requirements (if any), how did 
these change throughout the project, and what role did you play in shaping their expectations? 

3. Describe (if not already) the specific strategies used to obtain stakeholder buy-in in this project. 

4. Were the strategies effective? Flow do you gauge stakeholder buy-in? 

5. Were there situational or contextual variables that were different in the project that caused you to 
proceed in a certain way? 
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These questions were repeated in order to inquire about additional projects as time permitted. 



Analysis and Results 

All interviews were transcribed, and content analysis performed on the transcriptions. Participants 
described a range of strategies for obtaining stakeholder buy-in that they employed at different stages in the lifecycle 
of an instructional design project. First, a table of strategies will be presented, then elaborated upon in the context of 
the ADDIE model of instructional design. 

A list of strategies that were noted in interviews by at least two research subjects is shown in the table 
below. In general, throughout their interview, each participant returned to certain aspects of the instructional design 
process that they felt were most important for obtaining stakeholder buy-in. Interview questions were relatively 
open-ended, and the answers tended to reflect deeply held beliefs that that originate from the experiences of the 
participants. Therefore, in interpreting the chart below, the reader should consider the commonalities across different 
stakeholders rather than the exceptions. 

Table 1: Strategies Employed to Obtain Stakeholder Buy-In 



Strategy 


Subject 


Percentage Responding 


A 


B 


c 


D 


E 


Follow-up with stakeholders to determine satisfaction 
with intervention - job performance has 
improved/results achieved 


X 


X 


X 




X 


80% 


Establishing checkpoint reviews 


X 




X 


X 




60% 


Showing stakeholders several design options from 
which to choose 


X 






X 


X 


60% 


Design document 






X 


X 


X 


60% 


Pilot/acceptance phase 






X 


X 


X 


60% 


Building trust and building relationships 


X 


X 








40% 


Finding out what’s important to the client and designing 
to that 




X 






X 


40% 


Speaking in simple terms and proposing simple 
solutions, particularly in new client relationships 






X 


X 




40% 


Asking questions that respect the knowledge of the 
stakeholders 








X 


X 


40% 


Prototypes 






X 




X 


40% 


Using stakeholders in the creation of the instructional 
media 


X 








X 


40% 



Participants agreed that the strategies employed may vary depending on the situation. For example, work 
with first-time clients generally requires particular strategies to ensure buy-in that return clients do not require. Some 
clients, new or not, may not require as much involvement in the process, and some projects may be straightforward 
enough that they do not require as much buy-in as more complex or politically-sensitive projects. As for whether 
strategies varied according to setting (business, non-profit, government, or education), the subjects indicated that 
while differences might exist (for example, government work requiring formal documentation of the ADDIE 
approach, or a non-profit emphasis on low-cost solutions), the strategies they employed remained the same. 
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Instructional Design Process 

This section documents the picture of the instructional design process that emerged from data collection 
and analysis. 

Planning 

The subjects described an instructional design process, and a process of obtaining buy-in, that often began 
before the formal “Analysis” phase of the ADDIE model. For example, where a formal client relationship existed 
(where the participant was external to the organization for which they were working), some subjects placed an 
emphasis on setting expectations with key stakeholders through a formal written contract. They noted that setting 
and managing expectations was something that began prior to the start of a project but also continued throughout the 
project lifecycle. One participant noted, “I have to sell it before I can do it.” The participant was not excluding the 
need for a formal phase of analysis of the learners and their environment, but recognized that an up-front analysis 
was required to secure a contract and begin a more formal phase of analysis. In this respect, the ADDIE model of 
instructional design does not fully describe the process followed by experienced instructional designers in paid client 
relationships. What participants described is more akin to systems design, which begins with a “Planning” phase 
prior to analysis (see Valacich, George, and Hoffer (2001)). Therefore, the findings include several strategies for 
obtaining stakeholder buy-in that might first occur as part of a “Planning” phase, though they could also occur at any 
point during the design process. Each is described below. 

Speaking in Simple Terms 

Two of five participants spoke of the need to speak in simple, non-technical terms to key stakeholders. This 
meant avoiding jargon that might not be understood by those unfamiliar with instructional design. Subjects indicated 
that the need to speak in simple terms was a continual part of communicating with stakeholders, not limited to a 
particular stage of the instructional design process. 

Establishing checkpoint reviews 

Three of five subjects talked of conducting periodic, planned reviews of their work with relevant 
stakeholders throughout the lifecycle of the project. One noted that prior to an engagement she always established 
when reviews would take place and who would provide sign-off . She noted that this was important even as an 
internal consultant, since there were different expectations in an organization as to who would be involved in the 
review of materials, and at what points in the process. Another explained that he followed the ADDIE model, and at 
least between each stage provided a deliverable for sign-off by the stakeholders involved. In this way he could 
ensure that the final product would be accepted. A third subject said that he used Microsoft Project to map out his 
checkpoint reviews and check them off as they were done. 

Determining what is “important” to the client 

Two participants noted that finding out what is important to the client and designing a solution that takes 
that into account is essential to a successful engagement. One noted that a company’s mission statement was 
evidence of what the company felt important, and how the organization envisioned itself, and could therefore be 
used to develop an instructional solution that furthered that mission. A second participant echoed this sentiment in 
noting that a company’s mission, vision, and values were important to understanding what was important to the 
organization. He also observed that while a particular solution might be appropriate from an educational and ethical 
point of view, such a solution would not be successful if it did not reflect what the client held valuable. 

Building trust 

Building trust with the client was another strategy for obtaining buy-in that two of five participants noted. 
This was a continuous part of the instructional design process rather than a discrete event. Tactics included meeting 
expectations by delivering a quality product on time. One subject noted that building trust might also involve 
recommending someone else who was better suited for the work given the project requirements. He explained that 
clients remembered those who acted in their best interest, and would ask him back on future projects. In one 
situation the subject described, a salesman for his company had sold a customer training that the salesman claimed 
would change the culture of the customer’s company. When the subject entered the project kick-off meeting and 
learned of the unrealistic promise, he objected, and the customer called off the project. The subject also observed 
that building trust was a long-term strategy for obtaining stakeholder buy-in that could span multiple projects with 
the same client. 
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Analysis 



Asking questions that respect the knowledge of the stakeholders 

Asking basic questions (what one participant referred to as “dumb” questions) about the content domain or 
performance problem was a strategy for obtaining buy-in noted by two participants. One of the subjects, who had 
extensive experience with those who worked in the electrical construction industry, noted that “Stakeholders want 
you to recognize or help them express what they know is important.” 

Design 

Providing stakeholders with several possible design solutions from which to choose 

Providing stakeholders with several options for instructional approaches was a strategy noted by three of 
five subjects . One subject did not continue on a project without getting sign-off on a “proof of concept” - a visual 
representation of the instruction. She always provided three or more options for stakeholder(s) to choose among 
when providing such design options. A second subject noted that he always approached stakeholders with stock 
ideas he uses and asked them what they preferred. In this way, he explained, the stakeholder would feel that they had 
made a contribution to the project and feel ownership in it. 

Design Document 

A design document is a document that defines, among other things, the need, purpose, objectives, audience, 
instructional approach, and intended outcomes of the instruction. Three of five subjects noted that a design 
document was key to obtaining agreement with stakeholders as to what was important, the nature of the instructional 
intervention they would pursue, and the expected outcomes. 

Development 

Prototypes 

Two of five participants specifically noted that they created prototypes at the beginning of the development 
phase to allow stakeholders several different options from which to choose. This strategy is similar to that of 
providing several different design options during the Design phase. Both strategies are similar in that they provide 
stakeholders a choice, and therefore a sense of ownership in, the form that the ultimate deliverable will assume. 

Using stakeholders in the creation of the instructional media 

Two participants noted a strategy for obtaining buy-in specifically in the development of multi-media 
instruction. They asked stakeholders to play the role of significant characters within the instruction. This might be in 
the form of photographs, audio, and/or video. One subject noted that their direct involvement overcame resistance 
to the instruction, especially where technology-based training was new to the organization. 

Implementation 

Pilot Studies 

The use of a pilot study, beta test, or use acceptance phase prior to widespread implementation was 
commonly noted by several participants. One participant noted that she did pilot studies were the project was large 
or where there the training was computer-based, in which case a pilot might build acceptance as well as spread 
word-of-mouth about the course. Another noted that she tried to have actual learners review the content prior to 
delivering it in-class or online, but that in many cases her pilot study was the first instance of delivering the 
instruction. In such cases, she would frame the instruction as such and solicit feedback from the learners for 
adjustment of the next round of instruction. 

Evaluation 

Follow-up with stakeholders, and call-backs 

Four of five participants noted that they would follow-up with stakeholders following the intervention to 
gauge their satisfaction with the intervention. One described a phone call afterwards as “Sales 101,” and a means of 
developing and/or maintaining the relationship with key stakeholders. Since participants mostly worked in paid, 
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contract-based engagements, this was a strategy to ensure they maintained stakeholder buy-in even after the project 
concluded. One subject also described a focus-group approach in which he sat down with key stakeholders to 
discuss how they felt about the success of the project (or projects). Longer-term, when asked how they determined 
the success of the instructional intervention and/or obtaining stakeholder buy-in, participants noted that call-backs 
(i.e., repeat business) indicated previous success. 

Discussion and Conclusions 

The purpose of this research was to determine the strategies that experienced instructional designers use to 
obtain stakeholder buy-in. Prior research in this area has not concentrated specifically on this element of 
instructional design, though research has addressed practice generally. Visscher-Voerman (1999), as noted earlier, 
studied the practices of expert instructional designers, finding 16 design principles, and 4 specifically related to 
obtaining stakeholder buy-in: 

1) During the design process, designers should pay as much attention to creating ownership with clients 
and stakeholders, as to reaching theoretical or internal quality of design 

2) Designers should not only ask clients and (future) users for content-related input, but should also 
give them the right to decide about the design itself 

3) Designers should ask those with an important role in the development and implementation for their 
early participation in the design activity 

4) A useful means to help clients, partners, and other stakeholders to choose a solution and to formulate 
product specifications is by showing products from former projects. 

In addition, Pieters and Bergman (1995) and Kirschner, et al. (2002) note that instructional designers find it 
important to communicate with stakeholders to assess their receptivity to potential solution. These principles are 
congruent with the findings of this study, with one exception. With respect to showing products from former 
projects, several participants noted that they did not show projects from former projects due to client confidentiality 
agreements. Showing products from former projects was also a strategy that Kirschner et al. (2002) found as a 
practice among experienced instructional designers. The idea of providing stakeholders with several design options 
for comment on and selection from, however, was a similar strategy noted in this research. 

In this research, each subject approached the practice of instructional design from a different perspective, 
placing particular emphasis on certain elements that they found important based on their experience. While 
perspectives might have differed, however, the picture of the instructional design process that emerged from the 
interviews was one particularly oriented towards a few common approaches: 

• Setting and managing expectations with relevant stakeholders prior to and throughout the instructional 

design process by speaking in simple, easily understood terms and being realistic about the expected 

outcomes of the instruction 

• Determining what was important to stakeholders and designing a solution that addressed those needs 

• Conducting clearly defined checkpoint reviews throughout the instructional design process 

• Providing stakeholders with options to choose from, both during the design and development phase 

The most important point is that those interviewed saw obtaining stakeholder buy-in as a prerequisite to a 

successful instructional intervention. Without it, in other words, the intervention could not achieve it objectives. 
Reasons why this might be the case could vary, but could be because the critical knowledge had not been gleaned by 
asking questions of those who knew the material, or because those in an organization needed to champion the 
intervention and/or give it the necessary visibility had not been involved. The importance of achieving stakeholder 
buy-in to those interviewed indicates the value of this research, both for its intrinsic value in furthering academic 
discussion on this topic, as well as to practitioners who seek successful instructional design principles. 
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Abstract 

Learner-centered instructional design places students’ learning needs in the center of the instructional 
design process. In the learner-centered instructional design model for distance learning, learners’ motivational and 
affective needs are emphasized; steps of creating a learner-centered environment that supports learning strategies are 
described. This model provides a practical framework to guide the design of learning products for distance 
education that allow learners to actively construct knowledge and empower students to achieve learning objectives. 

Introduction 

Instructional design and learning are two different processes, but both relate to and influence each other. 
Instructional developers produce learning products, such as lessons, courses, and learning environments. Learners 
interact with learning products to acquire knowledge, skills, and competences (Jonassen, Davidson, Collins, 
Campbell, & Haag, 1995). The relationship between instructional design and learning is depicted in Figure 1. 
Learning products directly influence the effectiveness of learning. Different learning perspectives lead to different 
instructional design principles and processes. The instructional designer should be aware of the guidance of 
instructional design principles in the learning product design process. 




Instructional 
Design Process 




Learning 

Process 




Figure 1. The relationship between instructional design and learning. 




Figure 2. Teacher-centered instruction. 

In teacher-centered instruction, teachers control the teaching and learning process (Figure 2). Teachers 
select media and instructional strategies to present content information and motivate students to learn. An 
instructional product based on teacher-centered paradigm may benefit some learners but not others. This can be 
particularly challenging in developing products for distance learners, whose characteristics are diverse. They have 
different ages, interests, cultural backgrounds, technological skills, learning styles, and prior knowledge. 
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Figure 3. Learner-centered distance learning. 

In distance learning, it is better to allow students to control the learning process due to diversity of learner 
characteristics as well as time and locations. In learner-centered distance learning (Figure 3), students need 
motivation in learning and use learning strategies to interact with media that present learning content. Information 
may be pushed to students via media or students actively pull information. 

Learner-centered instructional design places students’ learning needs in the center of the instructional 
design process, and will more readily adapt to the diversity inherent in groups of distance learners. The principle of 
learner-centered instructional design is that the design is based on learner’s learning needs, and is for facilitating 
target learners to achieve learning objectives. The learner-centered instructional design paradigm focuses on the 
following issues. 

• Effectively motivate students to start learning, persist in the learning process, and continue to learn 

• Effectively present information (push and/or pull information) 

• Effectively provide and support students’ learning strategies 

• Effectively assess students’ learning outcomes 

A Learner-centered Instructional Design Model for Distance Learning 

The learner-centered instructional design model for distance learning includes five concurrent, overlapping, 
and recursive components: Understanding learning needs, Analyzing task and Developing learning objectives, 
Creating a learning environment. Developing learning assessment. Evaluating and revising learning product 
(UADCDE) (Figure 4). The UADCDE model specifically focuses on the process of designing lessons and courses 
for distance education. 

Understanding Learning Needs 

Understanding learner’s learning needs is a fundamental step for instructional designers to create effective 
and efficient learning products. In order to understand learning needs, it is necessary for instructional designers to 
analyze context as well as learner characteristics, motivational and affective needs, and learning strategies. 

One important issue in distance education is determining learner motivation. Different learners have 
different motivational needs while taking distance courses. Song (2000) identifies three types of motivation in web- 
based instruction: motivation to initiate, motivation to persist, and motivation to continue. Learners who take a 
distance education course need motivation to initiate their participation in the learning activities, persist in the 
learning process, and continue to take other courses after they finish one course. 

Learners’ affective needs are usually ignored in the instructional design process, partly because they are 
difficult to conceptualize and evaluate (Zvacek, 1991). However, learners’ affective states are closely related to their 
motivation, thereby influence learning. Thus, it is important for the designer to take these needs into account. One 
general affective need in distance learning is to know the instructor and other learners who are taking the same 
course. For example, learners in a course may want to know who the instructor is, what the instructor looks like, and 
the instructor’s qualifications for teaching the course. 
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Evaluate & Revise 




Figure 4. A learner-centered instructional design model for distance learning. 

Analyzing Task and Developing Objectives 

Instructional designers conduct task analysis in order to develop learning objectives, learning 
environments, and learner assessments. Learning objectives tell students what they need to learn and what they will 
be able to perform as a result of the learning process. 

Creating a Learning Environment 

A learning environment refers to “a place where people can draw upon resources to make sense out of 
things and construct meaningful solutions to problems” (Wilson, 1996, p. 3). A learner-centered environment 
enables students to construct meaning through their prior knowledge, beliefs, and cultural practices (Bransford, 
Brown, & Cocking, 2000). The design of a learning environment relates to learning objectives, learner 
characteristics, and learners’ needs. If motivational strategy and learning strategy support are embedded in content 
presentation, it will help students overcome barriers in their learning process. Figure 5 presents an example of 
combining content delivery with motivational strategy and learning strategy support. The creation of a learning 
environment for distance learning includes selection of delivery methods, selection of instructional methods, design 
of motivational strategies, design of learning activities, development of learning resources, and design of 
communication patterns. 
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Figure 5. An example of combining content delivery with motivational strategy and learning strategy support. 

Selection of delivery’ methods. Delivery systems are used to provide learning content and support 
communications in distance education settings. Delivery methods should facilitate achieving learning objectives, 
support communications, be appropriate for adapting to learners’ characteristics and needs, and be easy to set up and 
maintain (Mehrotra, Hollister, & McGahey, 2001). 

Delivery methods in distance education can be synchronous or asynchronous. In the synchronous mode, 
learning products are delivered to and received by the learners at the same time. Examples of synchronous delivery 
technologies for distance learning include radio and television broadcasts, two-way audio, and interactive television. 
In the asynchronous mode, the delivery and reception of learning products occur at different times. The common 
asynchronous delivery technologies for distance learning include printed materials, audio/video recording, and the 
Internet. 

Selection of instructional methods. Instructional methods are “strategies or techniques used to facilitate 
intended learning outcomes” (Head, Lockee, & Oliver, 2002, p. 262). Examples of instructional methods for 
distance learning include lecture, questioning, demonstration, discussion, group project, peer teaching, and role play. 
The selection of instructional methods in distance education should consider the learning objectives, learner 
characteristics, and delivery methods. 

Design of motivational strategies. One important aspect of learner-centered instructional design is learner 
motivation. Keller’s (1987a, 1987b, 1987c, 1999) Attention, Relevance, Confidence, and Satisfaction (ARCS) 
model and Wlodkowski’s (1999) time-continuum model are two practical frameworks that guide the systematic 
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process of motivational design and motivational strategy selection. Motivational and affective strategies used in 
distance learning environments should motivate learners to start learning in the environment, persist in the learning 
process, and continue to study another course after they finish one course (Song, 2000). 

Design of learning activities. Learning activities should be designed to encourage learners to actively 
participate in the learning process through meeting their motivational and affective needs and supporting their 
learning strategies. 

Development of learning resources. Learning resources are the primary sources of knowledge in distance 
learning. Examples of learning resources include instructional materials, discussion boards, and information on the 
Internet. 

Design of communication patterns. Interaction among learners and instructors is important for successful 
distance learning experience. Communication patterns can be synchronous (e.g., telephone, instant messenger, 
online chat, and videoconferencing) or asynchronous (e.g., email and discussion board). 

Developing Learning Assessment 

Assessment in education traditionally focuses on the evaluation of learners’ retention of knowledge and its 
applications in limited contexts (Reeves & Okey, 1996). The assessment in learner-centered learning environments 
emphasizes evaluating learners’ meaning-making process and performance. The development of learner assessment 
connects with the learning objectives, learner characteristics, learning environment, and student learning experience. 
Learning outcomes are directly measured with students’ work, such as portfolios, projects, and presentations. It is 
better to design multiple assessment formats to allow students reflect their multiple aspects of intelligence. 

Evaluating and Revising 

The evaluation in the UADCDE model includes formative and summative evaluation. The evaluation of 
learning products includes four themes. First, are learners’ learning needs completely understood and included in the 
learning product? Second, are learning objectives designed appropriately? Third, is the learning environment 
effective for facilitating learners’ knowledge construction? Fourth, is the learning assessment designed appropriately 
based on the learning objectives, learner characteristics, and the learning environment? 

Based on the feedback and evaluation results, all weaknesses found in the steps of understanding learning 
needs, analyzing task and developing objectives, creating a learning environment, and developing learning 
assessment will be revised to improve the quality of the learning product. 

Conclusion 

Learner-centered instructional design emphasizes meeting students’ learning needs in the learning product 
design process. In the learner-centered instructional design model for distance learning, learners’ motivational and 
affective needs are emphasized; steps of creating a learner-centered environment that supports learning strategies are 
described. This model provides a practical framework to guide the design of learning products for distance 
education that allow learners to actively construct knowledge and empower students to achieve learning objectives. 
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Abstract: In this paper we discuss how to use ontology to realize the course 
resources sharing in different E-learning systems. We elaborate the features of 
web course resources based on ontology. And according to the standard of 
resources development, we discuss the technology of constructing course 
resources ontology, and also provide an example of course resources ontology. 
Finally, a framework of Web course resources based on ontology is proposed. 
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Introduction 



The sharing of course resources is an effective way to improve E-learning quality, and reduce educational cost. 
China has the most educated people in the world, however, course resources distribute in a disproportion, which 
restricts educational popularization in some degree (Yang, 2005). The development of Web will promote course 
resources optimization distribution. So, Web course resources can narrow the gap. 

Nowadays, the Web is becoming the most popular educational medium in many fields, such as, at schools, 
universities, and for professional training. 

In fact, because of various factors, the sharing of Web course resources is not working very well (Ding, 2003). 
Investigations indicate that one main reason of it is that the norm and standard of terms are insufficient (Zhang 2004; 
Shi & Xiao, 2007). Different teachers describe the same term using different words, which lead to semantic 
isomerism as follows: 

• The Course resources of same subjects adopt different concepts and norms to describe; 

• The Same term expresses different meaning in different courses; 

• Different systems use different structure to same concepts; 

• Concepts in different systems exist various connections, but concepts can’t attain the mutual approbation 
within systems. 

Obviously, above factors make course resources difficult to share within different systems. Although there are initial 
attempts to develop some technologies for Web-based course resources sharing systems, these researches are still in 
an embryonic stage (Ding, 2003; Hou, 2004). 

In order to solve those problems, ontology (Studer, Benjamins & Fensel, 1998) can be used. Why did we use the 
ontology? Some of the reasons are: 

• To share common understanding of the structure of information among people or software agents; 

• To enable reuse of domain knowledge; 

• To make domain assumptions explicit; 

• To separate domain knowledge from the operational knowledge; 

• To analyze domain knowledge. 

Flowever, to make Web course resources reused is the main aim in this paper, ontologies include computer-usable 
definitions of basic concepts in the domain and the relationships among them. They encode knowledge in a domain 
and also knowledge that spans domains. In this way, they make that knowledge reusable. Ontology-based course 
resources are the basis of the hypermedia systems to the individual requirements of the learners and ontology-based 
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systems are very promising tools in the area of Web education: In the area of Web education it is important to take 
the different needs of learners into account in order to propose learning goals, learning paths, help students in 
orienting in the E-learning systems and support them during their learning progress. 

In this paper, we discuss how to construct domain ontology. We find ontologies with a significant degree of 
structure. These need to specify descriptions for the following kinds of concepts: 

• Classes (general things) in the many domains of interest; 

• The relationships that can exist among things; 

• The properties (or attributes) those things may have. 

Features of Web Course Resources Based on Ontology 

What are the benefits of the ontology-based web course resources? To sum up the features of ontology and web- 
based resources respectively, we can find that there are many advantages, such as authority, normativity, sharing, 
etc. 

Authority 

When web course resources are built based on ontology, the terms, definitions as well as relations of each domain 
are drew up by the domain experts. Digital resources developed from these terms and definitions will be authority in 
its domain. 

Normativity 

Ontology is built up according corresponding rules, usually using the standard language to describe the ontology. 
Now, the most recent development in standard ontology languages is OWL from the World Wide Web Consortium 
(W3C) (Horridge, Knublauch, Rector, Stevens, & Wroe, 2004). Therefore the description of the ontology is 
standard. Web course resources based on ontology will inherit the normativity. 

Sharing 

Ontology are used to denote domain knowledge which accepted by people. It reflects the recognition concepts 
collection. Ontology aims at an unit not the individual. So web course resources are built based on ontology will be 
reusable and sharing, concepts are easily accepted by scholars and teachers. 

Ontology of Web Course Resources 

To build ontology is a complex thing. Ontology is made up of a series of elements, each of which is composed of a 
kind of Relation and a series of related Concepts (Chandrasekaran, Josephson, & Benjamins, 1999). Through an 
ontology built by the teacher, it will be possible to describe the knowledge domain, the subjects constituting it, the 
relations among the various subjects, as well as methodologies and means with they are presented. 

In the following, we take the process of building ‘‘‘Educational Technology Ontology” (ETO)as an example, 
explained the methodology of developing ontology- based web course resources, and detailed several key 
technologies to realize how to share and reuse digital learning resources. 

General Development Methodology of Ontology 

There are many methods to develop ontology, such as Uschold Methodology, Gruninger & Fox Methodology, Meth 
Methodology, Knowledge-Engineering Methodology, etc (Staab, Schnurr, Studer, & Sure, 2001; Liu, Xue, & Wang, 
2007). Knowledge-Engineering Methodology is used more widely, but it will change according to various areas. It 
includes: 

• Identifying a purpose and scope. 

• Terms and concepts capture: Knowledge acquisition, listing important terms, concepts of different 
domains. 

• Building the framework of the ontology. 

• Integrating existing ontologies: Reuse of existing ontologies to speed up the development process of 
ontologies in the future. 
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• Ontology coding: Structuring of the domain knowledge in a conceptual model. 

• Evaluation: Verification and Validation. 

Build Web Course Resources Ontology 

• Limitation of general web course resources 

In general, web course resources are presented in the form of text information, multimedia, web pages and so on, 
which bring it hard to inter-manipulate and make it seriously duplicated construction. By establishing and executing 
educational resource norm, such as CELTS, SCORM, to some degree, the problem of sharing information 
exchanging formats between education platforms can be resolved. However, it didn’t provide the meaning of 
information. In other words, educational resource norm can only appoint the syntactic share of the exchanging 
recourses, but can’t appoint it’s sharing semantic. Therefore, it is especially important to rebuild web course 
resources. 

• Methodology of building web course resources ontology 

In fact, there is no single correct ontology-design methodology and we did not attempt to define one. The ideas that 
we present here are the ones that we found useful in our own ontology development experience. It mainly extracted 
from Knowledge-Engineering Methodology, and combined features of web course resources. 

Domain division: Divide domain according to different subjects, such as computer, educational technology, 
mathematical and so on. Develop domain ontology according to corresponding subjects, and build specialized 
ontology to describe the relation among different domain ontology. 

Terms collection: Gathered terms of different domain, well-understood domain inner concepts. The terms and 
concepts of digital learning recourses can be derived from experts, books, network, or even other existing ontology. 
Here we will use Dublin Core Meta Data Terms to help terms collection, the details will be explained later. 

Relation analysis: Ontology provides 4 basic relations: part-of kind-of instance-of and attribute - of (Noy, & 
McGuinness, 2001). These four basic relations cannot express all the relations, thus, it’s necessary to define 
relations according to specific requirement, all the relations should be expressed by accurate terms, and must not 
have ambiguous. 

Coding: Selected ontology language to describe ontology in order to be understood by machines. Common ontology 
languages include OWL, RDFS and so on. 

• The Dublin Core ontology 

The Dublin Core ontology is based on the Dublin Core Meta Data Terms (The full set of Dublin Core Meta Data 
Terms is described at http://www.dublincore.org/ documents/dcmi-terms/). The Dublin Core Meta Data Terms were 
standardized/ developed by The Dublin Core Meta Data Initiative (http://www.dublincore.org/). Dublin Core is a set 
of metadata elements that can be used to annotate various elements of an ontology with information such as 
‘creator’, ‘title’, ‘subject’, etc. So, Dublin Core Meta Data Terms are a set of elements/terms that can be used to 
describe resources — in our case, we can use these terms to describe the ‘resources’ such as classes, properties and 
individuals in an ontology, the following list contains a few examples: 

Title — Typically, a Title will be a name by which the resource is formally known. 

Creator — Examples of a Creator include a person, an organization, or a service. Typically, the name of a Creator 
should be used to indicate the entity. 

Subject — Typically, a Subject will be expressed as keywords, key phrases or classification codes that describe a 
topic of the resource. Recommended best practice is to select a value from a controlled vocabulary or formal 
classification scheme. 

Description — Description may include but is not limited to: an abstract, table of contents, reference to a graphical 
representation of content or a free-text account of the content. 

In order to annotate classes and other ontology entities with the above information and other Dublin Core Meta Data 
Terms the Dublin Core Meta Data ontology (DCOntology) must be imported. Protege has an automated mechanism 
for importing the Dublin Core Meta Data ontology. Through it we can import the Dublin Core Meta Data elements 
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Ontology. It will make us building ontology more easily. 

There are four types of basic relations to describe terms’ relations in an ontology, they are: Part-of Kind-of 
Instance-of and Attribute-of (Noy, & McGuinness, 2001). Part-of is used to express the concept relationship 
between part and all. Kind-of is used to express inheritance relations among concepts. Instance-of means the 
relations between instance and concept. Attribute-of means one concept is the attribute of another one. 

But only use the four basic relations mentioned above is not enough. There are many relations can’t be described by 
four basic relations. If we add some considers it will get a better solution. 

• Synonyms: For example, in Chinese, "computer" can also be called as "diannao". So in the ETO 
ontology, we use Synonym-of that we define by ourselves to define the relation above, and the bulk of 
unified words set can help resolving the problem. 

• The ambiguity of concepts: For example, "ATM" refers to a kind of network system in computer network 
domain. But it can also mean "Automatic Teller Machine" in bank domain. In order to remove the 
ambiguity, we can adopt domain division to resolve the problem. 

While using protege to build the ontology, we should do: 

• Determine the domain and scope of the ontology: In this research, we focus on Educational Technology 
domain. 

• Consider reusing existing ontologies: Through importing the Dublin Core ontology to reuse the Dublin 
Core ontology. 

• Enumerate important terms in the ontology: such as Distance Education, Instructional System Design 
(ISD), web, etc. 

• Define the classes and the class hierarchy 

• Define the properties of classes — slots 

• Define the facets of the slots 

• Create instances 

Following is part of ETO ontology: 

<?xml version-' 1 .0"?> 

<rdf:RDF 

xmlns:pub- 'http ://www.domain2. com#" 
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#" 
xmlns:owl="http://www.w3.org/2002/07/owl#" 
xmlns- 'http://www. owl-ontologies, com/unnamed, owl#" 
xml:base="http://www.owl-ontologies. com/unnamed. owl"> 

<owl:Ontology rdf:about=""/> 

<owl: Class rdf: about-'http ://www.domain2 .com#Person"/> 

<owl:Class rdf:about="http://www.domain2.com#Publication"/> 

<owl:Class rdf:about="http://www.domain2.com#Article"/> 

<owl: Obj ectProperty rdfiabout-'http ://www.domain2 .com#isCited"> 

<rdfs:domain rdfiresource- 'http://www.domain2. com# Article"/> 

<owl:inverseOf> 

<owl:Obj ectProperty rdf: about-'http ://www.domain2 .com#C iting"/> 

</owl:inverseOf> 

<rdfs:range rdf:resource="http://www.domain2.com#Article"/> 

</owl:ObjectProperty> 

<owl: Obj ectProperty rdfiabout-'http ://www.domain2 .com#hasAuthor"> 

<rdfs:range rdf:resource="http://www.domain2.com#Person"/> 

<rdfs:domain rdfiresource- 'http://www.domain2. com# Article"/> 

</owl:ObjectProperty> 

<owl:ObjectProperty rdf:about="http://www.domain2.com#isPublished"> 

<rdfs:range rdf:resource="http://www.domain2.com#Publication"/> 

<rdfs:domain rdfiresource- 'http://www.domain2. com# Article"/> 

</o w 1 : Obj e ctProp erty > 
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<owl:Obj ectProperty rdf:about="http ://www.domain2 . com#C iting"> 
<owl:inverseOf rdf:resource- 'http://www.domain2.com#isCited7> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article7> 

<rdfs:range rdf:resource="http://www.domain2.com#Article"/> 
</owl:ObjectProperty> 

<owl:DatatypeProperty rdf:about="http://www.domain2.com#keywords"> 
<rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#string7> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article7> 

</ ow 1 : Dat atyp eProp erty > 

<owl:DatatypeProperty rdf:about="http://www.domain2.com#hasTitle"> 
<rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#string"/> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article"/> 

</ ow 1 : Dat atyp eProp erty > 

<owl:DatatypeProperty rdf:about="http://www.domain2.com#content"> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article"/> 

<rdfs:range rdf:resource="http://www.w3.org/2001/XMLSchema#string"/> 

</ ow 1 : Dat atyp eProp erty > 

<owl : SymmetricProperty rdf: about="http ://www.domain2.com#bothC ited"> 
<owl:inverseOf rdf:resource="http://www.domain2.com#bothCited"/> 
<rdfs:range rdf:resource="http://www.domain2.com#Article"/> 

<rdf:type rdf:resource="http://www.w3.org/2002/07/owl#ObjectProperty"/> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article7> 

</owl : SymmetricProperty> 

<owl : SymmetricProperty rdf: about="http ://www.domain2.com#bothC iting"> 
<rdfs:domain rdf:resource="http://www.domain2.com#Article"/> 

<rdf:type rdf:resource- 'http://www.w3.Org/2002/07/owl#ObjectProperty7> 
<rdfs:range rdf:resource="http://www.domain2.com#Article"/> 

<owl:inverseOf rdf:resource="http://www.domain2.com#bothCiting"/> 

</owl : SymmetricProperty> 

<pub Publication rdf:ID="Info.Sci."/> 

<pub:Person rdf:ID="Keegan"/> 

<pub: Article rdf:ID="KB_757266_Individual_3"> 

<pub:hasTitle rdfdatatype- 'http://www.w3.Org/2001/XMLSchema#string" 
>Distance Education</pub:hasTitle> 

<pub:isPublished rdf: resource - "# I n to . Sc i . "/> 

<pub:keywords rdfdatatype- 'http://www.w3.Org/2001/XMLSchema#string" 
>Distance Education</pub:keywords> 

<pub:content rdfdatatype- ’http://www.w3.Org/2001/XMLSchema#string" 

> Distance Education is . . . ,</pub:content> 

<pub:Citing> 

<pub: Article rdf:ID="KB_757266_Individual_2"> 

<pub:hasTitle rdf:datatype="http://www.w3.org/2001/XMLSchema#string" 
>ISD</pub : hasTitle> 

<pub:isCited rdf:resource="#KB_757266_Individual_3"/> 

<pub:Citing rdf:resource="#KB_757266_Individual_3"/> 

<pub:isPublished rdf:resource="#Info.Sci."/> 

<pub : has Author> 

<pub:Person rdf:ID="Gagne"/> 

</pub:hasAuthor> 

<pub:content rdfdatatype- 'http://www.w3.Org/2001/XMLSchema#string" 
>ISD is ,...</pub:content> 

<pub:keywords rdfdatatype- 'http://www.w3.Org/2001/XMLSchema#string" 
>ISD</pub:keywords> 

</pub:Article> 

</pub:Citing> 

<pub:hasAuthor rdf:resource="#Keegan"/> 
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<pub:isCited rdf:resource="#KB_757266_Individual_2"/> 

</pub:Article> 

</rdf:RDF> 

Ontology Languages and Realization Tools 

There are many languages to describe ontology, such as Ontolingua, KIF, RDF, RDFS, OWL, etc (Corcho, 
Fernandez-Lopez, & Gomez-Perez, 2003). Different ontology languages provide different facilities. The most recent 
development in standard ontology languages is OWL from the World Wide Web Consortium (W3C). It can describe 
concepts and also provides new facilities. It also has a richer set of operators and negation. 

We can build on our experience using Protege (Noy, Fergerson, &Musen, 2000), WebODE (Arpirez, Corcho, 
Fernandez-Loopez, & Goomez-Peerez, 2001), and OntoEdit (Sure, Erdmann, Angele, Staab, Studer, & Wenke, 
2002) as ontology-editing environments. In this paper, we use Protege3.1 (new edition) to build ontology. 

A Prototype System Based on Ontology 



System Architecture 

In this research, we have designed the Ontology-based Web Course Resources Sharing System (OWCRS). It 
consists of three parts (as shown in Figure 1): Data layer, Middle ware layer and Application layer. 




Data layer: It realized the resources denotation which based on ontology. Using OWL fde format to describe web 
course resources, it contains rich and complex relations among different entities. So it provides models and methods 
of the organization, management, retrieval and inquiries for the knowledge. This is the basis of sharing resources, 
and is the core of the system. 
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Middle ware layer: Achieving OWL reasoning. The system will read OWL fde from the location where we have 
saved the ontology, and store in a specific model for further processing, then reason according to certain rules. This 
is a crucial step to realize the system. 

Application Layer: The interface that users access to web course resources. It mainly provides users with a friendly 
interface to achieve the right web course resources, and the future of resource sharing system. 

Some Key Techniques of the System 

Being based on a different logical model, we can use a reasoner (such as Racer) when build ontology. It can check 
whether or not all of the statements and definitions in the ontology are mutually consistent and can also recognize 
which concepts fit under which definitions. The reasoner can therefore help to maintain the hierarchy correctly. This 
is particularly useful when dealing with cases where classes can have more than one parent. 

When reasoning, we can use relations below: inclusion (c, d), equivalent (c, d) and disjoint (c, d). The most 
frequently used type inclusive relationship that is inclusion (c, d). When Class A contains Class B, it means that all 
examples in Class B must be examples of Class A. It can realize by ModelFactory.createOntologyModel or 
ReasonerRegistery.getOWLReasoner( ) to visit reasoner engine. For example: 

String resourceowl=”file: ETO-Owl”; 

Model schema=ModelLoader.loadModel(“resourceowl”); 

Reasoner reasoner=ReasonerRegistry.getOWLReasoner(); 

Reasoner - reasoner.bindSchema( schema); 

InfModel i nfmodcRModel Factory. create IntModelf reasoner. schema); 

Following is part of reasoning rules for resources citing relations in searching: 

bothCiting: (?a http://www.domain2.com#Citing ?c),(?b http://www.domain2.com#Citing ?c),notEqual(?a, 
?b)->(?a http://www.domain2.com#bothCiting ?b) 

bothCited: (?a http://www.domain2.com#Citing ?b),(?a http://www.domain2.com#Citing ?c)„notEqual(?b, 
?c)->(?b http://www.domain2.com#bothCited ?c) 

Realization of the System 

The system uses Eclipse, Myeclipse and Tomcat as a development environment, Java as the main developing 
language. Data layer make use of Protege3.1 ontology editing tools and use Knowledge-Engineering Methodology 
to build domain ontology, and save as OWL ontology files. Middle ware layer use semantic analysis to schedule all 
the components, functions and ontology. Jena 2.4 is used as a reasoner tool to reason OWL files and analyze 
resources. Application Layer is the main user interface and to show the reasoning result. 



Conclusions and Future Work 



In this paper, we presented the OWCRS system, which can realize web course resources sharing based on ontology. 
We firstly probe into the methodology of building web course resources ontology, using Artificial Intelligence 
research and internet technology for reference, and making full use of modem informational technology and 
international opening resources. Then we study the technology and method of web course resources sharing. But in 
practical, we faced many difficulties. Such as it depends on more manpower participated in while built the ontology. 
In the future, the research will focus on automatically separating and extracting resources. We believe that such a 
system would identify any referred object on the web, and its significant category and attribute -value information. 
Such a repository would be a valuable resource for scalable querying, integration and mediation over the web. 
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Storytelling is a good way to fulfill children’s happiness and enrich their imagination. The essential 
practical knowledge, social value, morality, and ceremonial in the human society can be learned by the children 
unobtrusively and imperceptibly. Storytelling is also a way to let children learn to interact with people such as 
discussing the story and telling the story to others. Moreover, children could promote their language ability by 
discussing and talking to others. 

Traditional storytelling is a “one-way, stable and abstract” communication between the storyteller and the 
children. The children listen to the story from storyteller and occasionally interact with storyteller. The story is 
usually from a book or what the storyteller heard before. The story will be told repeatedly with stable script and 
content without modification. Since the story is told by speech and voice, the children have to imagine and turn the 
abstract information about characters, scenes, objects, etc. into perceivable images in their brain. However, 
storytelling nowadays has become more sophisticated and become a “two-way, dynamic and diversified” 
communication. The children can listen to the story and play with the storyteller. Sometimes the storyteller may 
modify the script of the story depending on the feedback of the children. With digital technologies and electronic 
devices, diversified audio and visual content are added to enhance the effect of storytelling. 

It is essential for a storyteller to express the ambience of the story. With the aid of digital media, modern 
storytellers are able to attract the audience and make learning more interesting. Moreover, some storytellers build 
houses to incorporate storytelling with interactive technology. This new style of “house of storytelling” merged with 
digital environment and ambient space. To attract children to a house of storytelling, the space is designed with 
concepts extracted from computer games. There are concepts from hardware such as ambient stages, plentiful stage 
properties, spot light, music and sound effects and from software such asking the audience questions, let the 
audience make selection about the important events and repeating a scenario until the audience understand the 
consequence of the story. All the new concepts make the audience from passive to active. Information technologies 
such as virtual reality and augmented reality can be embedded in the storytelling space to create a space similar to a 
film of three dimensions. The audient is not merely “listen” to the story, they can “play” and “interact” with the 
story. Our goal is as far as possible to embody the content of stories. 

A prototype of interactive storytelling space to incorporate with the “two-way, dynamic and diversified” 
features extracted from computer games was developed to test the new style of storytelling. Digital content like 
animation and pictures were projected to the walls of the space. Music and sound effects were played by hidden 
stereo speakers to match up the movement of characters in projection. Several scenarios were designed to be 
interacted with the audience. No traditional interface such as mouse and keyboard were used to create an emerging 
ambient space. Computer vision technology was used to capture the image of the audience and detect their body 
movement. The audience could see themselves in the projected space and interacted with the virtual characters and 
scenes of the story. The interface of body movement is intuitive. The children did not need the knowledge of mouse 
control and keyboard input. They could just move their bodies, wave hands, stretch legs and any actions they like to 
play with the story. Since linear scenario and single ending of storytelling are traditional story structures, we 
considered that structure could be diversified for modern storytelling so that the audience was willing to the same 
story again and enjoyed the unpredictable and multiple results. Arborescent scenarios and endings were introduced 
into the story with key timestamps that the audience or storyteller could decide the preferred scenario. With the 
concept of interactive storytelling, the children become the main role who guides the development of the story. It 
was discovered that, when interactive technology was introduced, storytelling became more interactive and 
constructive. The children were more attracted by the dynamic and entertaining features of interaction. They were 
more active in the communication with the storyteller and other children. The learning also was more effective and 
less obstructive. Children are surely mankind’s future; however, nowadays parents have fewer children so that they 
are willing to let their children go to the house of storytelling even though it costs much money. Therefore, it is 
required to develop a better mold for storytelling and let children learn more happily. 
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Abstract 

First, this paper discusses the use of the Artificial Intelligence Markup Language (AIML) for the 
development of a conversational agent’s knowledge base. Second, this paper discusses two approaches to 
implementing conversational agents. One implementation option available to instructional designers is to run a 
conversational agent on a web hosting service capable of interpreting AIML tags. Alternatively, an instructional 
designer could develop a unique environment for implementation purposes. 

Introduction 

This work considers text-based conversational agents. Embodied conversational agents (Cassell, Sullivan, 
Prevost, & Churchill, 2000; Graesser, McNamara, & VanLehn, 2005; Ruttkay & Pelachaud, 2004) or, as Plantec 
(2004) describes them, V-people (virtual people) are animated characters capable of simulating human conversation. 
While animated characters provide greater novelty than textual delivery media (Clark & Choi, 2005), instructional 
effectiveness is of paramount importance to educators. 

Conversational agents offer instructional technologists an opportunity to design and to develop instruction 
that responds to student requests. Software that implements a conversational agent seeks to hold a seemingly 
intelligent conversation with the learner in order to help the learner meet instructional objectives. To hold an 
intelligent conversation, a software agent must access a knowledge base. First, this paper discusses the use of the 
Artificial Intelligence Markup Language (AIML) in order to create knowledge bases of conversational agents. 
Second, this paper contrasts two methods for implementing a conversational agent. Using the first method, 
instructional designers can create a knowledge base in AIML and import it to a web hosting service. As discussed in 
the first section of this paper, creating a knowledge base in AIML requires the inclusion of markup tags around 
conversational text. Also as noted below, this process has been automated. Consequently, an instructional designer 
can create the knowledge base of a conversational agent by writing question and answer sequences. The second 
approach requires computer programming, which is more complex than creating AIML and importing the 
knowledge base to a hosting service. Although the second approach is more complex, creating a custom made 
conversation permits instructional designers to implement unique features not available in hosting environments. 

Using AIML to Develop the Knowledge Base of a Conversational Agent 

The Artificial Intelligent Markup Language (AIML) was developed by Richard Wallace (see 
http://www.pandorabots.com/pandora/pics/wallaceaimltutorial.html) and conforms to the rules of the Extensible 
Markup Language (XML). As such, AIML features tags similar to ones in XML and other language derivatives, 
such as the Hypertext Markup Language (HTML) and Voice XML. Accordingly, familiarity with HTML or XML 
would be helpful when learning AIML, but such knowledge is not required. Although AIML is not a part of the 
work of the World Wide Web consortium, AIML provides an XML-compliant standard for the creation of a 
knowledge base. The latest version of AIML (version 1.0.1), which became stable in March 2005, contains 40 tags. 
For a complete list of the tags, see http://www.alicebot.org/documentation/ptags.html. Discussion of all AIML tags 
is well beyond the scope of this paper. Nevertheless, the subset of AIML tags illustrated in the following examples 
serves to highlight key features of the markup language. 

One can construct a knowledge base using three basic AIML tags: <category>, <pattem>, and <template>. 
In AIML, knowledge is represented within the <category> and </category> tags and knowledge is conceived of as a 
linguistic stimulus-response pair. The stimulus appears within <pattern> and </pattern> tags and the response 
appears within <template> and </template> tags. Whenever the words within the <pattern> tags are encountered, 
the conversational agent responds with the words within the <template> tags. The following example is a complete 
AIML script capable of responding only to the question: Where is Anaheim? 
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<?xml version-' 1.0" encoding="UTF-8"?> 

<aiml version="1.0"> 

<category> 

<pattern>WHERE IS ANAHEIM</pattern> 
<template> Anaheim is in California</template> 
</category> 

</aiml> 



In the preceding script, the first statement is required for implementation purposes in order to convey that 
the tags in the script are XML-compliant. The second line identifies the script more specifically as containing 
AIML tags. The script, which is a strikingly simple conversational agent, can be executed at the following URL. 
http ://www.pandorabots.com/pandora/talk?botid=a03 50a2c3 e3 6fe9e 

Keep in mind that you will receive the response only when you ask the question: Where is Anaheim? Those three 
words must be spelled correctly. The letters of those words may be entered in any combination of uppercase and 
lowercase letters. Entering the question mark is optional. 

The following AIML example could be used to tutor novice computer users who have yet to learn how to 
open application programs. In order for a human tutor to help such novice users, the tutors must be able to 
remember what was said recently. This is also true of conversational agents implemented through computer 
software. In AIML, the <that> tag enables a conversational agent to recall its last utterance. This is illustrated in the 
following example. 



<?xml version- T.0" encoding="UTF-8"?> 

<aiml version-' 1.0"> 

<category> 

<pattern>HOW DO I OPEN MICROSOFT WORD</pattern> 

<template>On what operating system do you want to run word</template> 

</category> 

<category> 

<pattern>HOW DO I OPEN MICROSOFT EXCEL</pattern> 

<template>On what operating system do you want to run excel</template> 

</category> 

<category> 

<pattern>MAC OS *</pattern> 

<that>On what operating system do you want to run word</that> 

<template>Click the Word icon in the palette</template> 

</category> 

<category> 

<pattern>MAC OS *</pattern> 

<that>On what operating system do you want to run excel</that> 

<template>Click the Excel icon in the palette</template> 

</category> 

<category> 

<pattern> WINDOWS *</pattern> 

<that>On what operating system do you want to run word</that> 

<template>You may be able to double click a Word icon on the desktop; otherwise, proceed through the 
Start button, to Program Files, to Microsoft Office, to Word</template> 

</category> 

<category> 

<pattern> WINDOWS *</pattern> 

<that>On what operating system do you want to run excel</that> 

<template>You may be able to double click an Excel icon on the desktop; otherwise, proceed through the 
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Start button, to Program Files, to Microsoft Office, to Excel</template> 
</category> 

</aiml> 



The preceding example can be executed at the following URL: 
http ://www.pandorabots.com/pandora/talk?botid=be 1 e lc203 e3 6fe9 1 

This example demonstrates that all conversational transactions in an AIML knowledge base appear within the 
<category> . . . </category> pair of tags. Second, note that the asterisk (*) is a wildcard character that matches any 
input. Consequently, in response to the operating system question posed by the conversational agent, the user can 
enter Mac OS X or Windows 95, Windows XP, or any other text after either MAC OS or Windows and a match will 
be found. Then, the conversational agent’s reply will depend on whether the agent previously asked about running 
Word or Excel. Whichever operating system and application pair is of concern, the response in the appropriate 
<template> . . . </template> tags will be displayed. For instance, if the agent asked: “On what operating system do 
you want to run excel” and the user responded, Windows XP, the agent would reply with: “You may be able to 
double click an Excel icon on the desktop; otherwise proceed through the Start button, to Program Files, to 
Microsoft Office, to Excel.” 

The use of numerous categories in the previous example is inefficient because the procedure for opening 
Microsoft applications is uniform. Indeed, even though only the name of the application changes, a new category 
must be inserted into the AIML for each new Microsoft application. To eliminate this inefficiency, a variable can be 
assigned the name of the application entered by the user. This is shown in the following example. 

<?xml version-' 1.0" encoding="UTF-8"?> 

<aiml version-' 1.0"> 



<category> 

<pattern>HOW DO I OPEN MICROSOFT *</pattern> 

<template>On what operating system do you want to run <set 
name= " app " ><star/></set></temp 1 ate> 

</category> 

<category> 

<pattern>MAC OS *</pattern> 

<that>On what operating system do you want to run *</that> 

<template>Click the <get name="app"/> icon in the dock.</template> 

</category> 

<category> 

<pattem> WINDOWS *</pattem> 

<that>On what operating system do you want to run *</that> 

<template>You may be able to double click the <get name="app"/> icon on the desktop; 
otherwise, proceed through the Start button, to Program Files, to Microsoft Office, to <get 
name="app"/>.</template> 

</category> 



</aiml> 



This example can be executed at the following URL: 
http://www.pandorabots.com/pandora/talk7botkU8ee67b9bde36fe90 

In AIML, the <star/> tag contains the text associated with the asterisk wildcard character. Consequently, in the 
script above, the <star/> tag will contain the name of the Microsoft application the user wants to open when the user 
asks a question like “How do I open Microsoft Outlook.” The agent uses the name of the application entered by the 
user to pose the operating system question. This could have been accomplished with the following AIML: 

<template>On what operating system do you want to run <star/></template> 
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For the script to function properly, however, the name of the Microsoft application entered by the user must be 
saved in a variable. This is accomplished in the example above by enclosing the star tag in a set tag, as in <set 
name="app"><star/></set>. The use of a variable is vital because the initial value of the star tag (i.e., the name of 
the Microsoft application entered by the user) is overwritten when the user responds to the operating system 
question. Once the user has responded to the operating system question, the conversational agent responds with 
suitable instructions for opening the application. Notice that the get tag (i.e., <get name="app"/>) is used to display 
the name of the Microsoft application entered by the user. 

Other AIML tags provide conversational agents with additional capabilities, including, but not limited to 
the ability to: reply randomly to user input; respond to grammatical variations expressing the same utterance; assign 
a gender to the agent; and to assign a name to the agent. For a tutorial on the tags that enable these and other 
features, see http://www.pandorabots.com/botmaster/en/tutorial. 

Implementing a Conversational Agent 

One may implement a conversational agent through a free hosting service. Alternatively, one may develop 
a unique environment. Both approaches are considered here. 

Agent Hosting Services 

Anyone with a knowledge base that conforms to the AIML specification can implement their 
conversational agent, which may also be called a hot, at a free hosting service. Pandorabots 

(http://www.pandorabots.com) is an example of such a free service and was used to implement the three examples in 
the previous section. 

To implement conversational agents at pandorabots one first creates an account by supplying one’s first 
name, last name, email address, and password. Once logged in, a conversational agent can be created in three steps. 
First, click the link called Create a Pandorabot ; supply a name for the hot and if you have your own AIML, select 
the last radio button, which starts the bot with no initial content or knowledge. Second, click the AIML link in the 
top menu bar. At this point you may upload an AIML file or click the link called Create a new AIML file. If you 
take the latter option, then copy and paste your AIML categories into the form; supply a file name and click the web 
form’s submit button, which is labeled Save as. After the uploading or copying-and-pasting step, the last step is to 
publish the agent: Click the My Pandorabots link; click the radio button beside the name of the bot you just created; 
ensure that Publish is selected in the drop-down menu; and click the Go button. This completes the bot 
implementation process by displaying the hyperlink to your conversational agent. 

Even without AIML, one can create a conversational agent at pandorabots with conversational text like the 
following four lines. 

Hello, how are you? 

Hi, I am fine. 

Where is Anaheim? 

Anaheim is in California. 

In this case, pandorabots creates AIML categories, complete with pattern and template tags. To experience this 
conversion, you could enter the four lines above, or some other conversational text in which each utterance is 
separated by a blank line, into the form at http://www.pandorabots.com/botmaster/en/aiml-converter.html and then 
click the convert button. 

Another possibility for hosting a conversational agent involves the installation of free AIML processing 
software on a personal computer. Since the source code is often available, this approach creates the possibility of 
extending the system in order to provide unique features. To pursue this implementation option, see the installation 
details at http://www.alicebot.org/downloads/programs.html or http://www.alicebot.org/aimlbots.html. 
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Custom Implementation 

Creating a custom agent requires skill in computer programming whereas implementing an agent in AIML 
demands only the inclusion of category, pattern and template tags around conversational text. This is the biggest 
difference in the two approaches and a considerable disadvantage of custom development. However, in addition to 
holding a conversation, a custom agent can be programmed to provide additional services. For example, a 
conversational agent acting as a tutor may be able to diagnose learner errors and provide feedback intended to help 
learners correct their errors. This is precisely the type of functionality this author sought to provide to students 
learning how to create web pages in order to fulfill the portfolio requirement of a technology teacher preparation 
course. 

By writing unique code in PHP (a hypertext preprocessing language), the author has created and 
implemented A Web Tutor (A.W.T. or, more simply, AWT) that converses with students in order to help them 
create web pages. In addition to providing answers to questions about web page development, AWT provides 
lessons that often include step-by-step instructions. Further, AWT provides diagnostic and feedback features that 
help students identify “broken hyperlink'’ and “image not found” errors in their web pages. One may perceive of the 
tutor’s conversational, tutorial, diagnostic and feedback capabilities as a kaleidoscope, capable of offering unique 
experiences to individual learners within a domain of practice. 

Learners communicate with AWT by typing a question into a text field on a web page. Specifically, 
learners can ask AWT the following questions: 

• What is all the fuss about the World Wide Web? 

• What is a web browser? 

• What is a web server? 

• What is HTML? 

• What is a web page? 

• What is a home page? 

• What is a URL? 

• What is a web site? 

• What is a web crawler? 

• What is a search engine? 

• What does FTP mean? 

In addition to asking those questions, a learner can ask AWT the following questions in order to leam how 
to complete various tasks concerning web page development. 

• How do I access my portfolio web space? 

• How do I create a web page? 

• How do I publish a web page? 

• How do I edit a web page in Netscape Composer? 

• In Netscape Composer, how do I insert a hyperlink? 

With respect to diagnosing any errors in the addresses of images and relative hyperlinks, learners can enter 
“check my web page” or “check my web page named xxxx.yyy” (where xxxx can be any file name and yyy is either 
htm or html). Upon encountering either of those directives, AWT retrieves the particular web page (either 
index.html or the specific page identified in the directive); detects any errors in the relative hyperlinks and in the 
references to images by parsing the HTML of the web page and retrieving the file names of all files in all folders of 
the student’s web space; and provides feedback. The feedback identifies any errors in the URLs of the images and 
relative hyperlinks; states the cause of each error; and provides a remedy to fix the error. Given the feedback, the 
learner can ask additional questions, which will prompt AWT to provide step-by-step instructions for performing the 
remedies. 

The conversational capabilities of AWT are limited at this time, but continued development is planned in 
light of current research. Future versions of the software will improve upon three main aspects of the system, 
particularly: (1) The tutor’s conversational ability, which will enable the tutor to reply to more questions; (2) The 
tutor’s knowledge base, which will permit the tutor to provide additional lessons and will permit each learner to try 
alternative approaches if the learner is not satisfied with the initial set of instructions; and (3) The tutor’s interface, 
which will send the tutor’s output through a text-to-speech engine to an animated character in order for the tutor’s 
comments to be rendered as voice output through an avatar. 
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Abstract 

This research study investigated the relationship of two types of visual presentations and their effects on college 
students’ cognitive load performance and knowledge transfer. 1 12 graduate students enrolled in an introductory 
statistical course and an educational research course at a mid-west university participated in the study. The research 
design incorporated animated visual display and static visual display of a statistical concept as instructional message 
design strategies with controlling for prior knowledge and spatial ability. Results indicated that students in the 
animation group had significantly higher knowledge transfer score than those in the static visual group. Students in 
the animation group also had significant higher mental efficiency score than those in the static visual group. 

Introduction 

In instructional design technology, inconclusive results have been obtained in animation research studies 
(Large, 1996; Park & Hopkins, 1993; Rieber, 1990). Positive learning effects have been found in some studies 
(Atkinson, 2002; Blankenship & Dansereau, 2000; Craig, Gholson & Driscoll, 2002; Mayer, 1997; Mayer & 
Anderson, 1991; Mayer & Moreno, 2000; Mayer & Sims, 1994; Mayer & Anderson, 1992; Park, 1998; Park & 
Gittelman, 1992). However, no significant learning effects have been found in other studies (Ausman, Lin, Kidwai, 
Munyofu, Swain & Dwyer, 2004; Rieber, Boyce & Assad, 1990; Rieber, 1996; Rieber & Hannafin, 1988; Rieber, 
1989; Zhu & Grabowski, 2004). Controversy still exists about whether animation assists in learning or produces no 
effect. Large (1996) points out that the results of animation studies have been “contradictory and inconclusive” (p. 
9). According to Ausman et al. (2004), without careful consideration of the message and systematic design, 
animation may do “more harm than good” (p. 49). It is apparent that there is a need to further investigate the effect 
of animated instructional messages to provide learners with an instructional environment where concept acquisition 
and retention can be effectively transferred and meaningfully connected to their cognitive loads. 

Related research and theory 

The present study adopted dual coding theory, cognitive load theory, and multimedia design model to 
construct the design experiment. Paivio’s dual coding theory has been widely used in multimedia instructional 
design (Mayer & Anderson, 1992; Mayer & Moreno, 1998; Mayer & Moreno, 2002; Mayer, 1997). The theory 
believes that cognition consists of two systems: one is the nonverbal system, which represents and processes visual 
image information; the other one is the verbal system, which represents and processes language information. Both 
systems are separate but can be referentially connected (Sadoski, Paivio & Goetz, 1991). 

The two systems have different organizational characteristics. The verbal system processes information in 
sequence, but the nonverbal system processes information in a synchronous or parallel manner (Paivio, 1986). The 
cognitive activity starts when the sensory system finds either verbal or nonverbal stimuli. Representational 
connections are made to find corresponding systems. These representations are formed hierarchically in the system. 
In the verbal system, representations are formed in a sequential and logical order. In the nonverbal system, 
representations are formed in a holistic way. For example, the eyes, nose, and mouth can be viewed separately but 
are usually viewed as parts of the face (Sadoki, Paivio, & Goetz, 1991). See Figure 1 for reference. 
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Figure 1. Dual Coding Theory 



Dual coding theory: Verbal and nonverbal representation systems 
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Associative connection refers to the organization within a system. For example, a spoon can make you 
think of an entire table setting. Referential connections refer to inter- system relations. For example, when the word 
table is said, you can picture a table in your mind. Dual coding theory emphasizes both associative processing and 
referential processing (Sadoki, Paivio, & Goetz, 1991). 

Dual coding theory provides this research study with a theoretical framework. Based on Paivio’s dual 
coding theory and Mayer’s multimedia learning model, the program design will use both verbal and imagery 
systems. The purpose is to optimize the sensory registry and make sure that the extraneous cognitive load is low; 
therefore, learning will be enhanced. 

Cognitive load theory identifies three kinds of cognitive loads. One is intrinsic cognitive load; one is 
extraneous cognitive load, and another one is germane cognitive load. Intrinsic cognitive load means that either the 
learning material has a lot of elements to assimilate or that learning these elements involves interactivity. Extraneous 
cognitive load refers to the instructional technique (Sweller & Chandler, 1994). Germane cognitive load refers to the 
effects of variability in materials presented to learners (van Merrienboer & Sweller, 2005). 

According to cognitive load theorists, a human’s short-term memory is extremely limited, while long-term 
memory has a huge capacity for storing information. Building schema automation is a very crucial step for learning. 
If automation recognition occurs, the learning process will bypass the working memory and go directly to the long- 
term memory to retrieve information. In this way, the working memory capacity will be freed to allow other 
cognitive activities (Sweller & Chandler, 1994). 

Cognitive load theory provides researchers with an instrument to measure mental load from a task-based 
perspective and mental effort from a learner-based perspective (Sweller, van Merrienboer, & Paas, 1998). Mental 
load can be measured by using the number and interactivity of learning elements, while mental effort can be 
measured by using subjective, physiological and performance-based indices (Sweller, van Merrienboer, & Paas, 
1998). 

Richard Mayer has been researching whether combining different instructional strategies with animation 
can produce better learning outcomes. His research shows that providing students with only verbal explanations will 
not help students learn scientific concepts. In order to achieve the best results in learning scientific concepts, 
multimedia learning models need to be provided. Mayer’s research studies further demonstrate that prior knowledge 
and spatial ability can influence the learning outcomes (Mayer, 1997). If prior knowledge and spatial ability can 
influence the learning outcomes of utilizing a multimedia learning model, what will happen if we control these 
variables, and will learning outcomes differ? Mayer’s research helps to form the research questions and the research 
design, which will use both animation and narration of the text in order not to increase the cognitive load and 
optimize selective perception function. 
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Purpose of the study 

The purpose of study was to exam the effects on students’ cognitive load performance and knowledge 
transfer by types of visual presentations. The first focus of the design was to measure whether animation and static 
visual display created different amounts of mental load and whether learners exposed to different instructional 
strategies would impose different mental efforts on performance. The second focus of the design was to measure 
whether animation and static visual display could improve knowledge transfer ability. To test these assumptions, 
the following research questions were proposed: 

• Does the type of instructional strategy affect performance on knowledge transfer tests, controlling for prior 
knowledge, visualization and spatial orientation? 

• Are the mean scores on mental efficiency the same or different for students in the two instructional 
condition groups? 

The null hypotheses are that there is no difference on the mean scores on knowledge transfer tests for 
students in the two instructional condition groups, controlling for prior knowledge, visualization and spatial 
orientation. There are no differences on the mean scores on mental efficiency for students in the two instructional 
groups. The alternative hypotheses are that there is difference on the mean scores on knowledge transfer tests for 
students in the two instructional condition groups, controlling for prior knowledge, visualization and spatial 
orientation. There are differences on the mean scores on mental efficiency for students in the two instructional 
groups. 



Research method 

The study was a quasi-experimental design using analysis of covariance (ANCOVA) to test knowledge 
transfer. ANCOVA analysis tests whether the population means on the dependent variable vary across levels of a 
factor, controlling for covariates (Green & Salkind, 2005). In this study, two groups were compared. One was 
animation group. The other was static visual display group. One dependent variable, knowledge transfer test scores 
were compared. Three covariates were prior knowledge, visualization and spatial orientation. The independent 
variable was instructional strategy which included two levels with one being the animation and another being the 
static images. Animation is generally defined as a dynamic visual display and static images is generally defined as 
static visual display (Anglin et al., 1996; Park, 1998). 

Participants 

The participants were 112 graduate students enrolled in an introductory statistical course and an 
educational research course at a mid-west university. Participants were randomly assigned to two different 
instructional strategies: an animated visual display group and a static visual display group. Sixty participants served 
in the animation group and fifty two participants served in the static visual display group. 

Procedures 

Prior to the experiment, each participant was tested on prior knowledge on sampling distribution, spatial 
orientation and visualization. Participants were then randomly assigned to either a computer program that used 
animated instruction, or another computer program that used static visual instruction. Upon completion of the 
instruction, participants took knowledge transfer tests and reported their mental effort score. 

Computer Programs 

The computer program for animation was designed and developed by Rice University Statistics Lab and the 
researcher. The animation program used Java Applet. The researcher used Adobe Captivate 2.0 to create the 
interface and the knowledge transfer test. The computer program for static visual display was designed by the 
researcher by using Adobe Captivate 2.0. The concept to be presented in the programs was that of the sampling 
distribution of means in an entry level graduate statistics course and educational research course. The animation 
program allowed subjects to pick the number of observations and the number of the samples. Based on the subject 
input, the program generated the concept of random sampling. Static visual display program presented the key 
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frames of sampling distribution. Previous research had shown the positive effect of adding narration into the 
software so that cognitive load would be reduced and students can combine verbal and visual systems together based 
on working memory theory and dual coding theory (Mayer & Moreno, 1998; Mayer & Sims, 1994; Mayer & 
Anderson, 1992; Paivio, 1986). Narration was used in both animation and static display groups in this research 
study. 

Scoring 



The primary author scored all the paper tests. There were five questions in the prior knowledge test, and 
each question weighted 2 points. There were altogether 40 questions in the spatial orientation test. Each question 
weighted one point and the total was 40 points. There were 5 questions in the visualization test, and each weighted 1 
point. The total was 5 points. Knowledge transfer test had five questions and each weighted 2 points. The software 
was programmed to return the score when all questions got answered. 

Mental efficiency score was used to test the mental effort subjects used to understand the concept. 

Cognitive mental load is generally defined as the number and the interactivity of the learning elements (Sweller, van 
Merrienboer, & Paas, 1998). Mental effort is defined as “the amount of cognitive capacity or resources that is 
allocated to accommodate the task demand” (Sweller, van Merrienboer, & Paas, 1998, p. 266). The focus of the 
study is to see whether animation and static visual display create different amounts of mental load, whether learners 
exposed to different instructional strategies will impose different mental efforts on performance, and whether 
different instructional strategies will help students apply what they have learned into real life situation. 

Spatial orientation and visualization test 

The spatial orientation and visualization test designed by Educational Testing Service was used to gather 
information for two of the controlling variables, spatial orientation and visualization. The test of spatial orientation 
was testing mental rotation, while the visualization assessed both mental rotation and the ability of performing serial 
operations. Both visualization and spatial orientation require mental rotation in short-term memory, but visualization 
requires more component of performing spatial operations (Ekstrom, French, Flarman, Dermen, 1976). 

According to Mayer and Sims (1994), spatial visualization was the most relevant aspect to learning from 
animations, so only spatial orientation and visualization tests will be used to examine students’ spatial ability in this 
research. In spatial orientation test, students were asked to decide whether each of the cards on the right was 
identical or different from the card at the left. All of these drawings are of the same card, which has been slid around 
into different positions on the page. If the card cannot be made to look like sliding it around on the page, then 
choose D (different), otherwise, choose S (same). See Figure 2 for reference. For the visualization test, some figures 
were displayed at the left of a line, and others displayed at the right of the line. Figures at the left represent a piece of 
paper being folded, and one or two circles on the paper indicating where the paper was punched. Figures at the right 
represent where the holes would be when the paper was unfolded. In visualization test, students was asked to decide 
which one of the figures at the right showed where the holes would be when the paper was unfolded (Ekstrom, 
French, Flarman, Dermen, 1976). See Figure 3 for reference. Spatial orientation and visualization test was 
administered as a pencil and paper test. The test will be distributed to the subjects before the computer programs. 



Figure 2. Card Rotation Test 
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Figure 3. Paper Folding Test 
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Mental load and mental effort test 

Mental effort score was gathered at the end of the intervention. Students reported the mental effort they 
used when either computer program is presented to them to understand the concept of sampling distribution. The 
mental load and mental effort test was based on Borg ’s study on aspects of mental load (1978). After receiving the 
computer instruction, each participant reported their mental effort score. The score was designed to gather 
information on the effort used to solve a specific task. The procedure was based on the assumption that the subjects 
can introspect on their cognitive processes and report how much mental effort they spend on each task (Marcus, 
Cooper, Sweller, 1996). A 7 - point self report scale was used to measure students’ perception of amount of mental 
effort. This 7 - point response scale is designed by Nadine Marcus, Martin Cooper, and John Sweller. The scale 
categories are (1) very easy, (2) easy, (3) fairly easy, (4) neither easy nor difficult, (5) fairly difficult, (6) difficult, 
and (7) very difficult (Marcus, Cooper, & Sweller, 1996). 

Reliability of the knowledge transfer tests was constructed. The Cronbach’s alpha value for the knowledge 
transfer test is medium with a value of 0.4. Based on the assumption that the data are multidimensional, factor 
analysis is performed and the results review that question two and five are measuring the same latent construct, 
question three and four are measuring the same latent construct, and question 1 are measuring one latent construct. 
All rotated factor loadings are high for all the similar constructs. The results confirm that the reliability for questions 
two and five are high, the reliability for questions three and four are also high. 

Prior knowledge test and knowledge transfer test 

The prior knowledge test measures how much statistical information subjects have already grasped before 
being exposed to the intervention. The knowledge transfer test measures whether subjects can apply the knowledge 
they have just learned into solving a real life statistical problem. Both tests are statistical exams. Prior knowledge 
test was administered as a pencil and paper test. Knowledge transfer test was imbedded into the computer software. 

Results 

A one-way ANCOVA assesses whether population means on the dependent variable are the same across 
levels of a factor, adjusting for differences on the covariate (Green & Salkind, 2005). This research study tests 
whether population means on the dependent variable knowledge transfer test are the same across levels of a factor, 
adjusting for differences on the covariates prior knowledge, spatial orientation, and visualization tests. 

Four assumptions are needed for the ANCOVA test. Assumption 1: the dependent variable is normally 
distributed in the population for any value of the covariate and for any one level of a factor (Green & Salkind, 2005). 
This assumption is met. Assumption 2: the variances of the dependent variable for the conditional distributions 
described in assumption 1 are equal (Green & Salkind, 2005). Levene’s test of equality of error variances is 
performed and result p = 0.003 is significant, indicating that the variances are not equal. The possible explanation 
for this result is that the group sample sizes differ. Animation group has 60 participants joined the study and static 
visual group has 52 participants joined the study. Assumption 3: the cases represent a random sample from the 
population, and the scores on the dependent variable are independent of each other (Green & Salkind, 2005). All the 
subjects participated into the study are graduate students who registered educational statistics I course and 
educational research course. They are randomly selected from the population. Assumption 4: the covariate is linearly 
related to the dependent variable within all levels of the factor, and the weights or slopes relating the covariate to the 
dependent variable are equal across all levels of the factor. This assumption is also called as homogeneity-of-slopes 
assumption (Green & Salkind, 2005). Homogeneity-of-slopes assumption has been tested. The test evaluates the 
interaction between the covariate and the factor in the prediction of the dependent variable knowledge transfer test. 
The interaction source is labeled group * prior knowledge, group * spatial orientation, and group * visualization. 

The results suggest the interaction for group * prior knowledge is not significant, F (2, 105) = 3. 1 84, p = 0.045, 

partial T] = 0.057. The results suggest the interaction for group * spatial orientation is not significant, F (2, 105) = 
2.5 12, p = 0.086, partial T] = 0.046. The results suggest the interaction for group * visualization is significant, F (2, 

105) = 6.008,/? = 0.003, partial T] = 0.103. Non-significant results will assume homogeneity of slopes. Two of the 
interaction effects are not significant with moderate effect size. One interaction effect is significant with moderate 
effect size. The ANCOVA test is proceeded assuming homogeneity of slopes. 
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The results of the ANCOVA analysis indicate that the null hypothesis should be rejected, F (1, 107) = 

2 

21.240 ,p < 0.01, and the partial Tj~ of 0.166 suggests a strong relationship animation group has on knowledge 
transfer tests, controlling for prior knowledge, spatial orientation, and visualization tests. The relationship between 
the prior knowledge and knowledge transfer test is not significant, F (1, 107) = 0.669, p >0.05, with the covariate 

accounting for about 0.6% ( the partial lj of 0.006) of variance of the knowledge transfer test for animation group. 
In this study, the relationship between the spatial orientation and knowledge transfer test is not significant, F (1, 

107) = 0.668, p >0.05, with the covariate accounting for about 0.6% (the partial of 0.006) of variance of the 

knowledge transfer test for animation group. The relationship between the visualization and knowledge transfer test 
is significant, F 107) = 5.431,/? <0.05, with the covariate accounting for about 4.8% (the partial of 0.048) of 
variance of the knowledge transfer test for animation group. 

The correlation results show that there is moderate correlation exists between knowledge transfer and 

2 

visualization, with r = 0.207 and p < 0.05. It can be concluded that 4% of the variance (.207 " ) of the visualization 
is accounted for by its linear relationship with knowledge transfer. There is low correlation exists between 
knowledge transfer and spatial orientation, with r = 0.123 and p > 0.05. There is low correlation exists between 
knowledge transfer and prior knowledge, with r = 0.029 and p > 0.05. 

Mental efficiency score is obtained by using the formula: 

T7 _M~P 

'll ' 

The formula is introduced by Paas and van Merrienboer. M is the mean performance scores and P is the mean metal 
effort scores. For this study, the performance scores are the knowledge transfer test scores and mental effort score 
has been obtained by asking subjects to report the mental effort they used to understand the concept of sampling 
distribution by the help of animation or static visual displays (Pass, Tuovinen, Tabbers, & Gerven, 2003; Paas & 
Merrienboer, 1994). The rating scale is designed by Nadine Marcus, Martin Cooper, and John Sweller. The 
numerical values and associated labels range from very easy (1) to very’ difficult (7). 

For static visual group, mean performance scores on knowledge transfer test is 4.34, and mean mental 
effort score is 5.86. 



F _ M ~P _ 4 . 34 - 5.86 

V2 



-1.075. 



For animation display group, mean performance scores on knowledge transfer test is 6.37, and mean mental 
effort score is 5.41. 



M-P _ 6 . 37 - 5.41 

42 = 42 



0 . 68 . 



Figure 5. Mental efficiency as a function of knowledge transfer and mental effort. Animation group represents high 
efficiency and static visual group represents low efficiency. 



270 





The data obtained from the present study indicated that animation group outperformed static visual group 

2 

on knowledge transfer test (p < 0.001). The partial T] of 0.166 also suggested a strong relationship animation 
group has on knowledge transfer tests. This finding further supports previous researches conducted on the effect of 
animation and knowledge transfer (Mayer & Moreno, 2000; Mayer & Sims, 1994; Mayer & Anderson, 1992; 

Mayer, 1997; Craig, Gholson, & Driscoll, 2002). 

A number of research studies demonstrated that animation produced no effects on learning (Ausman, Lin, 
Kidwai, Munyofu, Swain, & Dwyer, 2004; Rieber & Hannafin, 1988; Rieber, 1989; Rieber, Boyce, & Assad, 1990). 
This research study stated that animation group has mean score of 6.47 on knowledge transfer test, and static visual 
group has a mean score of 4.25. Animation group outperformed static visual group in 2.2 units on knowledge 
transfer test. 

For students’ prior knowledge, the present study indicated that there is low correlation between knowledge 
transfer test score and prior knowledge score. This finding suggests that students with high prior knowledge do not 
necessarily outperform students with low prior knowledge. This is incongruent with the previous research conducted 
by Schnotz and Rasch (2005) that for learners with prior knowledge, they needed external support from animation, 
though their research results proved that students with low prior knowledge performed better on test scores after 
learning with static pictures than animation. 

For students’ spatial orientation, the results showed that interaction for group and spatial orientation was 
not significant, but the interaction for group and visualization was significant. The relationship between the spatial 
orientation and knowledge transfer test was not significant with the spatial orientation accounting for 0.6% of 
variance of the knowledge transfer test for animation group. The relationship between visualization and knowledge 
transfer test was significant with the visualization accounting for about 4.8% of variance of the knowledge transfer 
test for animation group. The results suggested that visualization is correlated with animation learning which 
confirmed Mayer’s research on students’ spatial ability and their information processing in memory (Mayer, 1997). 

For students’ mental load, the mental efficiency score for animation group was 0.68, slightly above the E 
=0 line as shown in Figure 5, indicating that animation group is of high efficiency, with high performance score and 
comparatively low mental effort score. Static visual group had mental efficiency score -1.075, indicating low mental 
efficiency with low performance score but high mental effort score. The results confirmed that for learning involves 
spatial visualization; animation would help knowledge transfer and improve mental efficiency. 
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A Quest for Instructional Design Competencies, Methods, and Tools to 
Support Effective Performance Assessment 



Tamme E. McCowin 
IP A Research Foundation 

In today’s fast-paced, transient, global economy, instructional technology (IT) professionals have no way to 
manage and track performance to industry competency standards. An IT professional’s work life is constantly 
changing and new methods, tools, and technologies have an immediate impact on individual development. To 
effectively perform the functions of the instructional designer and developer roles, in any work environment, 
practitioners need the ability to measure and monitor their performance on industry defined competency standards. 
This would enable them to assess their strengths and weaknesses on core competency standards. IT professionals 
“have a responsibility to keep their skills current” (Rothwell & Kazanas, 2004, p. 386). Richey, Fields, Foxon, 
Roberts, Spannaus, and Spector (2001), claimed updating and improving one’s knowledge, skills, and abilities is an 
important and essential competency. With the right measurement methods and tools professionals would be 
empowered to measure, score, and monitor their own performance on existing competency standards. This would 
also enable them to make effective decisions about their professional development and career planning activities. 
Similarly, employers and educational organizations would also be able to measure and monitor individual 
performance for recruitment, selection, placement, succession planning, training, development, and career 
counseling. A literature review was conducted to identify what competencies, methods, and tools are extant in the 
field to enable effective performance assessment of IT professionals. The author will discuss the research problem 
that led to the literature search, identify prior research studies, review the literature, and explain the solution. This 
literature review was conducted as apart of the author’s dissertation research. In-depth coverage of the literature 
review and instrument development and validation results can be found in the dissertation. 

Problem Statement 

Instructional designers and developers often fulfill multiple roles across IT domains (Seels & Richey, 

2001). Business and industry perpetuates this practice requiring that professionals assume multiple roles throughout 
the training and development process. Carrying more than one role or wearing multiple hats at the same time has 
become common practice in the field. “When a situation calls for it, the professional slips out of one role and ‘puts 
on’ another. . . because a vast body of underlying skills and knowledge supports their execution.” (Bernthal, 
Colteryahn, Davis, Naughton, Rothwell, & Wellins, 2004, p. xxiii). At the same time, "the [IT] profession. . .[has 
become] more complex and sophisticated. . .[and this] leads to specialization" (Richey et al., 2001, p. 107). This 
form of specialization is one of synergy and integration not segmentation and discord. Nor does it imply the notion 
of a widely renowned axiom jack of all trades. Rather, it suggests that those professionals who embrace the 
challenge and see themselves as integrated professionals with multiple competencies, talents, and skills across IT 
domains instead of specialized professionals in a single domain will possess a competitive advantage over those 
individuals who do not. 

What is more, effective job performance can be mitigated when no means is available to accurately and 
objectively assess performance. To address this problem a literature review was conducted to identify prior research 
undertaken to answer the research question: what are the valid and reliable competencies, methods, and tools for 
assessing the preparation and performance of IT professionals? This literature review sought to synthesize into a 
cohesive whole the various theories, practices, methods, competency standards, and tools extant in the field to 
support the development and validation of the integrated performance assessment (IPA) methodology. By 
developing the IPA methodology, professionals, employers, and educational organizations will have an accurate and 
objective way to assess individual performance on known industry competency standards. 

Literature Search 

To answer the research question an extensive literature search was conducted using literary databases: 
Psychlnfo, ERIC, Proquest, Questia, EBSCOHost, Proquest Dissertations and Theses, Internet search engines, 
books, Performance Improvement Quarterly (PIQ) Journal, and Educational Technology Research and Development 
(ETR&D) Journal. Several search terms and categories were identified. Instructional design was selected as the 
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knowledge domain of interest with a focus on competencies, measurements, expertise, expert practices, processes, 
methodologies, models, performance assessments, and scale development. Results from Questia produced no 
studies, results from Psychlnfo, Proquest, and EBSCOHost produced nine studies, results from ERIC produced two 
studies, results from Proquest Dissertations and Theses database produced five studies, results from ETR&D 
produced four studies, and results from PIQ produced six studies. Competency standards were retrieved from 
publications issued by the International Board of Standards for Training, Performance and Instruction (IBSTPI) and 
National Workforce Center for Emerging Technologies (NWCET). Table 1 shows the distribution of references by 
literary source. Of the 104 references identified for the entire study 40% were within the past ten years and 29% 
were within the past five years. 



Chart 1: Number of References 
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A total of 18 studies were identified within the general search term categories related to instructional design 
competency and the acquisition of expertise in instructional design. Table 2 illustrates the distribution of results 
obtained from the literature search by study type, which were deemed most critical and specific to the research 
question. Eleven qualitative studies related to ISD practice were conducted to identify and describe the differences 
between novice and expert instructional designers, practices employed during normal training and development 
activities, and skill classifications of competencies (see Larson & Lockee, 2004; Visscher-Voerman & Gustafson, 
2004; Lui, Gibby, Quiros, & Demp, 2002; Reiser, 2001a; Reiser 2001b; Lui & Hempstreet, 1998; Perez & Emery, 
1995; Winer & Vazquez-Abad, 1995; Wedman & Tessmer, 1993; Rowland, 1992, Gayeski, 1991; and Atichson, 
1996). Four studies related to instructional design competency were conducted to define and establish knowledge 
domains, taxonomies, and standards (see NWCET, 2003; Richey et ,al, 2001; Song, 1998; Atchison, 1996). 



Chart 2: Number of Research Studies 




Competency Definition (CD), Model Development (MD), Tool Development (TD) 



One study was conducted to determine the relationship between ISD models and multiple intelligences 
(Tracey & Richey, 2007). One study was conducted to develop a measurement instrument that could discriminate 
between masters and non-masters of instructional design (Stepp, 1995). Another study was conducted to develop 
and validate a multiple intelligences measurement instrument. Finally, one study was conducted to compare skill 
competencies with Bloom’s taxonomy (Ven & Chuang, 2005). Of the studies found none were conducted to 
establish criterion and predictive validity and reliability of ISD competencies as a basis for assessing individual 
performance. In addition to these 18 studies, it was necessary to include seminal works from cognitive psychology, 
psychometrics, systems theory, and espoused theory (theory-in-use) to prepare the literature review. 
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Review of the Literature 



Eighteen studies were conducted by various researchers and theorists in the field that met the search 
criteria. Many of these studies set the foundation of research in instructional design practice for such areas as 
competency definition, model development, tool development, and comparative analysis. The studies described in 
this section represent seminal works that support the recommended solution. 

ISD Practice Studies 

There was a wealth of literature and theory extant in the field that described and prescribed how to design 
instructional solutions, few studies looked at what IT professionals did in practice. One study conducted by Rowland 
(1990) showed that novice and expert designers differ in the way they carried out instructional design process steps 
and the types of solutions they devised. Another study conducted by Wedman and Tessmer (1993) sought to identify 
the frequency with which IT professionals used or omitted instructional design process steps in their projects. The 
study identified leading factors contributing to an IT professional’s selection of certain design activities. 

Winer and Vazquez-Abad (1995) replicated Wedman and Tessmer’s 1993 study. The results of this study 
showed that individuals typically entered the field through education technology programs, on-the-job-training, 
personal or professional contacts, training seminars and workshops, and the International Society of Performance 
Improvement (ISPI). The homogeneity of the results also confirmed the validity of the ISD process used in Wedman 
and Tessmers’s study. The results “allow us to make some general statements about what instructional designers do, 
and why they do not perform certain steps” (Winer & Vazquez-Abad, 1995, p. 63). A second and equally important 
benefit of this study indicated that selection of instructional strategies and media selection were the most important 
aspects of ISD, and these would undoubtedly have an impact on ISD processes in the future. This also suggested that 
IT professionals would “have to expand their focus [and skill repertoire] to include systemic influences and cultural 
constraints in the creation of learner-centered learning environments” (Winer & Vasquez-Abad, 1995, p. 65). In 
looking to the future, Winer and Vasquez-Abad (1995) suggested that these results make one wonder whether 
professionals would continue to use ISD models, and whether those models would continue to be appropriate tools 
to use to create learning and performance solutions. 

A study conducted by Lui, Gibby, Quiros, and Demp (2002) showed how the field has diversified and 
grown in complexity, and new media design and development is a new reality for IT professionals. New trends and 
technologies have impelled IT professionals to acquire a multiple set of skills to cope with the increased demands in 
the industry. As the knowledge for building robust and innovative products and delivering rapid solutions continue, 
IT professionals have found themselves engulfed in more than one type of project. They must assume roles in areas 
previously reserved for more specialized professionals. Oftentimes, they may even find themselves wearing more 
than one hat or carrying more than one role (Bernthal et al., 2004; Larson & Lockee, 2004; Lui et ah, 2002). In fact, 
“the term [instructional design and instructional developer] is less familiar outside the field. Instead, one hears job 
titles such as industrial designer, curriculum developer, [e-]learning specialist, instructional technologist, [subject 
matter expert, and] sometimes just project manager” (Lui et ah, 2002, p. 2). This suggests that the title instructional 
designer or instructional developer may no longer be an appropriate title for every professional in the field; this is 
most noticeable due to the ever-changing new media tools, technologies, and delivery mediums. To be more 
consistent with emerging trends, it may be more appropriate to refer to IT professionals as new media integrators, 
especially since job titles, roles, responsibilities, and the work products they produce are a result of applying an 
interspersed skill set. 

Competency’ Definition Studies 

Atchison (1996) conducted a qualitative study to identify the competencies of expert IDs. Atchison’s study 
helped to further classify, expand, and better define the 1986 IBSTPI standards (Atchison, 1996). The results from 
Atchison’s study clearly explicated the differences between novice and expert practice. Song (1998) took the next 
step in helping to validate both the 1986 IBSTPI standards and Atchison’s expert competencies. Song’s study sought 
to determine whether IT professionals in the field could further classify the complexities of the competencies 
(Richey et al., 2001; Song, 1998). Song (1998) used a descriptive research method and developed a survey 
instalment using both the 1986 IBSTPI standards and Atchison’s expert competencies. She was able to further 
classify the competencies at the novice, intermediate, and expert levels. 
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IBSTPI used the findings from the Atchison and Song studies to develop a newer and broader set of ISD 
competencies - the 2000 IBSTPI standards. These newer competency standards reflect current practices in the field. 
IBSTPI conducted a study to validate the new competencies and performance statements for use in the profession 
(Richey et al., 2001). This validation study used two survey instalments: one to measure designer perception of 
competency criticality and the other to determine expertise levels required on-the-job to demonstrate each skill 
(Richey et ah, 2001). IBSTPI also created two skill classifications for the standards: essential and advanced. 

While Atchison (1996) was quantifying the ISD exemplary competencies at the expert level, the NWCET 
identified eight information technology skill standards (NWCET, 2003). Around the same time that Song (1998) 
was conducting her study, NWCET also conducted nationwide research to validate the information technology skill 
standards. NWCET identified and updated the skill standards with new and emerging workforce job roles, technical 
knowledge, and related foundational skills (NWCET, 2003). Three career clusters comprised of 18 competencies 
were identified for the instructional design role and several key job roles required of IT professionals were also 
included in job classifications for information technology professionals. 

These four studies were conducted to identify, classify, and define existing skill standards. The Atchison 
study was a qualitative study to further expand known ISD competency standards at the expert level. The study 
focused heavily on clearly explicating a dividing line between novice and expert practice. Similarly, the Song (1998) 
study sought to further establish differences in expertise at three different skill levels: novice, intermediate, and 
expert. The 2000 IBSTPI study established a new set of ISD competencies, which validated the content of each 
competency and performance statement and ultimately classified each competency as either essential or advanced. 
The 2003 NWCET standards further expanded ISD knowledge domains and competencies within the information 
technology field. None of the four studies established the criterion and predictive validity or reliability of the ISD 
competency standards. Although the independent research efforts of IBSTPI and NWCET creates a professional 
bridge between instructional technology and information technology fields. 

Model Development Studies 

One study was conducted to develop and validate a multiple intelligences (MI) design model (Tracey & 
Richey, 2007; Tracey, 2001). The original study was completed as dissertation research and published in the 
ETR&D journal. This research sought to integrate the theoretical foundations of instructional design with multiple 
intelligences into one model that can be used by instructional designers to guide the creation and development of 
instructional materials to enable learners to construct meaningful learning in multiple ways (Tracey & Richey, 

2007). The study also had the added goal to establish the validity of the MI model using internal validity procedures 
(Tracey & Richey, 2007). The results from this study confirmed the validity of a combined model that uses the 
tenets of instructional design and multiple intelligences to guide the design and construction of learning. 

Tool Development Studies 

Stepp (1990) conducted research to validate a measurement instrument to discriminate between masters and 
non-masters of instructional design using the IBSTPI standards as a framework. Stepp’s final paper and pencil test 
instalment consisted of 50 test items for the original item bank. Content review was conducted to establish validity 
with two subject matter experts and the final item bank consisted of 35 items. This instrument was administered to 
257 subjects. Eighty-three subjects participated in item analysis and the remaining 184 participated in instrument 
validation, which resulted in a Pearson point-biserial coefficient of .695 for validity and a Cronbach alpha of .746 
for reliability (Stepp, 1990). Discriminate analysis of the instrument showed that all but 4 questions were successful 
discriminating between masters and non-masters. Omission of these four questions increased the phi coefficient to 
.758 and the Cronbach alpha coefficient increased to .762 (Stepp, 1990). 

Several conclusions and recommendations were made to further this research effort and improve upon the 
approach. First, comments made by many masters during the study, suggested that Stepp’s instrument should be 
used more as a research tool rather than a certification tool. Although Stepp used a norm-referenced approach in his 
overall design, it was limited by his extrapolation and omission of certain competencies that represent higher 
ordered thinking, interaction, and complexity. The decision was made to focus on developing a norm-referenced 
instalment, which would not require all competencies to distinguish between masters and non-masters. This was 
due, in part, to the fact that Song believed that certain standard competencies was immeasurable through structured 
and objective test methods (Stepp, 1990; Reynolds et al., 2006; Schwurith et al., 2005). Performance assessment 
methods such as direct observations and portfolios are alternative means that can be used to ascertain higher levels 
of skill capability. Reynolds et al. (2006) posited, “performance assessments require test takers to complete a 
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process or produce a product in a context that closely resembles real-life situations” (p. 239). Second, further 
research is required to reinvigorate and expand upon the efforts made in Stepp’s study. An extended research study 
should focus on extending Stepp’s study to look at the entire set of IBSTPI and NWCET standards using the self- 
rater, criterion-referenced, and performance assessment methods. 

Other Comparative Studies 

A study conducted by Ven and Chuang (2005) was completed to determine the classification of action 
verbs for competency standards in the information science profession in three different countries: America, 
Australia, and Taiwan. An action verb lexicon was developed to correctly classify action verbs using Bloom’s 
taxonomy for the cognitive domain. A total of 341 action verbs were identified and categorized. Chart 3 illustrates 
the distribution of action verbs according to Bloom’s Taxonomy (Ven & Chuang, 2005). 




This literature review was conducted to identify prior research undertaken to answer the research question: 
what are the valid and reliable competencies, methods, and tools for assessing the preparation and performance of IT 
professionals? The search also sought to verify and validate the need to develop and validate the IPA research 
methodology. The IPA research methodology (see Figure 1) consists of four studies: the ISD Performance Inventory 
study, Multiple Intelligences ISD (MIISD) Construct Map study, Integrated Skills Assessment (ISA) study, and IPA 
study. 
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Figure 1: IPA Research Methodology 



Study one is the focus of the research dissertation 




RQ 

What are the valid and 
reliable competencies 
for assessing the 
preparation and 
performance of IT 
professionals? 



RQ1 

What is the relationship 
between Ml Constructs 
anc ISD competencies? 

RQ2 

What combination of Ml 
Constructs and ISD 
Competencies influence 
an IT professional's skill 
integration? 

RQ3 

What combination of Ml 
Constructs and ISD 
Competencies influence 
an IT professional's skill 
imbalance? 



HI 

IT professionals who are 
able to correctly respond 
to a large number of 
competency-intelligence 
cluster items will exhibit 
skill integration. 

H2 

IT professionals who are 
not able to correctly 
respond to a large 
number of competency- 
intelligence cluster items 
will exhibit skill imbalance. 



RQ 

What is the 
relationship between 
perceived, assessed, 
and demonstrated 
performance? 



ISD Performance Inventory’ Study 



This first study will 1) establish a framework for scoring IT professionals across all IT domains and related 
disciplines on known competency standards, 2) better classify and explicate ISD competencies to reflect stages of 
growth and development using the Dreyfus model (Dreyfus & Dreyfus, 1995), and 3) expand the validity and 
reliability of ISD competency standards through quantitative analysis. IBSTPI and NWCET sought to validate 
similar but discrete competency standards for IT professionals. IBSTPI developed and validated a set of 23 
competencies across four separate knowledge domains (Richey et al., 2001). NWCET developed and validated a set 
of 18 competencies across three career clusters (NWCET, 2003). The ISD Performance Inventory study will be 
conducted to develop a valid measurement instrument based on the combined IBSTPI and NWCET standards. The 
domains, competencies, and performance statements identified and refined in previous studies (see NWCET, 2003; 
Richey et ah, 2001; Song, 1998; Atchison, 1996) will serve as a framework that the researcher used to create the 
initial item pool for the ISD Performance Inventory. Additionally, the Dreyfus model consists of five stages: novice, 
advanced beginner, competent, proficient, and expert. The researcher used the Dreyfus model as the rating scale for 
each item in the initial item pool for the instrument. Finally, development of the final ISD Performance Inventory 
depends on establishing the validity and reliability of the instrument through quantitative analysis. To accomplish 
the goals of this study the researcher will use a four-step scale development and validation process. This process was 
proven to be the best empirical method for developing and validating measurements (Viswanathan, 2005; DeVellis, 
2003; Netemeyer, Bearden & Sharma, 2003). 



MIISD Construct Map Study 



Study two will be a cross-validation of ISD competencies and MI constructs to identify competency- 
intelligence relationships between these two mutually exclusive components. The identified competency-intelligence 
clusters will describe how these constructs influence skill integration and skill imbalance. This study seeks to answer 
the following: What is the relationship between MI constructs and ISD competencies? What combination of MI 
constructs and ISD competencies influence an IT professional’s skill integration? What combination of MI 
constructs and ISD competencies influence an IT professional’s skill imbalance? To answer these questions it is 
necessary to first establish the relationship between intelligence and competency. The researcher will obtain 
intelligence and competency scores using two separate measurements. The Multiple Intelligence Developmental 
Assessment Scales (MIDAS) and the ISD Performance inventory will serve as measurement instruments. The 
MIDAS is a 106-item report completed by an individual or a knowledgeable informant (Shearer, 1996). The MIDAS 
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measures a person’s intellectual capabilities on all MI constructs (Shearer, 1996). Similarly, the purpose of the ISD 
Performance inventory is to measure a person’s skill capabilities on ISD competencies. Analysis of the data from 
both measures will help to determine the correct classifications and descriptions of competency-intelligence clusters. 
Once the number of competency-intelligence clusters has been identified, the information may be used to describe a 
person’s skill imbalance or skill integration. 

ISA Scale Study 

Study three will be to develop and validate the ISA scale. The ISA scale is a criterion-referenced 
measurement based on the competency standards and competency-intelligence clusters defined in studies 1 and 2. 
This study seeks to test the following hypotheses: IT professionals who are able to correctly respond to a large 
number of competency-intelligence items will exhibit skill integration. IT professionals who are not able to correctly 
respond to a large number of competency-intelligence items will exhibit skill imbalance. To test these hypotheses it 
is necessary to overcome the subjective limitations of self-reporting measures and 360-degree reviews. A better and 
more objective performance assessment method is attainable through criterion-referenced testing. A criterion- 
referenced test measures what a person knows or can do compared to what he or she must be able to know or do in 
order to perform a job or task successfully (Reynolds et al., 2006; Swezey, 1981). Criterion-referenced tests are 
designed to measure a person’s skill capabilities against known performance standards. In the case of the IT 
professional these known performance standards include the 2000 IBSTPI and 2003 NWCET standards. Stepp 
(1990) developed a measure that could discriminate between IT professionals. Stepp’s final instrument was norm- 
referenced and consisted of 50 test items for the original item bank in paper and pencil format. Further research is 
required to reinvigorate and expand upon the efforts made by Stepp. An extended research study should 1) focus on 
replicating Stepp’s study, 2) focus on a criterion-referenced approach to look at the entire set of IBSTPI and 
NWCET standards, and 3) broaden the scope of subject groups used for the study. The ISA study offers an 
alternative method to extend past research efforts. 

IP A Study 

Study four will be to answer the research question: What is the relationship between perceived, assessed, 
and demonstrated performance? This study will consist of three parts and employs a concurrent triangulation 
strategy (mixed methodology) using the case method. “Concurrent procedures, . . .[in mixed-methods studies,] 
collect both forms of data at the same time during the study and then integrates the information in the interpretation 
of the overall results” (Creswell, 2003, p. 16). In this case, a combined quantitative and qualitative approach 
mitigates the limitations of a single approach while canceling out the biases of either approach (Creswell, 2003). 

Part one of this study will ask participants to rate themselves on the ISD Performance inventory. Part two will ask 
participants to complete the ISA scale. Part three will ask participants to complete an ISD project to measure their 
skill capabilities through product development, observations, and interviews. The data from these three measures 
provide a snapshot of an IT professional’s skill capabilities, and may be used to develop a skill capabilities profile 
for an individual. 

Theoretical Framework 

The entire IPA research methodology focuses on four theoretical areas: multiple intelligence theory, 
systems theory, psychometric theory, and espoused theory (theory-in-use). Multiple intelligence theory describes the 
connection between intelligence and competency. There is a neurological connection between competence and 
intelligence and identification of competency-intelligence clusters will provide a basis for measuring both constructs 
(Connell, Sheridan, & Gardner, 2003). According to Fodor (1983): 

The mental causation of behavior typically involves the simultaneous activity of a variety of distinct 

psychological mechanisms, [and] the best research strategy would seem to be divide and conquer: first 

study the intrinsic characteristics of each of the presumed faculties, then study the ways in which [those 

faculties] interact, (p. 1) 

Systems theory provides a theoretical explanation of the interactions between competence, intelligence, and 
environmental factors. This notion is an outgrowth of Cartesian theory, which explicates that cognition, is directly 
linked to human behavior and consists of modular and integrative faculties (Connell et al., 2003; Fodor, 2000; 

Fodor, 1983). 



280 




Psychometric theory describes the relationship between variables to determine which combination of 
competency-intelligence clusters describes and influences skill integration and skill imbalance. Connell et al. (2003, 
p. 1 36) claimed “it is possible to parse the space of human cognitive capacities in many ways.” The studies outlined 
here represent one possible method for parsing intelligence and competence in order to describe the skill capabilities 
of IT professionals. Skill integration and skill imbalance is central to the entire IPA research methodology. As an IT 
professional uses one or more of his or her intellectual faculties (MI constructs), this builds behavioral skill 
capacities in multiple content domains or disciplines (ISD competencies), which can lead to either skill imbalance or 
skill integration. Skill imbalance is the natural tendency to overuse or concentrate on one or more closely related 
competency-intelligence clusters, which causes skill-lopsidedness and inflexibility because certain competency- 
intelligence clusters are over or under used. This behavior leads to skill imbalance because it is easy for individuals 
to want to exercise or improve their skills in those areas that they enjoy or have a natural affinity towards. Skill 
integration, on the contrary, is the ability to combine and use a mixture or blend of MI constructs and ISD 
competencies. This behavior results in versatility and flexibility. However, it is difficult to accomplish skill 
integration because it requires making a conscious effort to use all one’s skill capabilities, which may be dormant 
and requires development, under developed, or under used due to lack of practice. 

Espoused theory helps to explain perceived, assessed, and demonstrated performance. What someone 
thinks he or she is capable of doing, how others know that he or she is capable, and how that capability may be 
demonstrated are not always in alignment. Schdn (1983) suggested, “every competent practitioner can recognize 
phenomena - families of symptoms associated with a particular. . .[issue, problem, or situation] - for which he [or 
she] cannot give a reasonably accurate or complete description [but recognizes that he or she has some level of 
competency or ability poised ready to solve it]” (p. 49). 

Each theory is fundamental to the IPA research methodology. First, the ISD Performance inventory study 
uses espoused theory and psychometric theory as the theoretical framework for data collection and analysis. Second, 
the MIISD construct map study and ISA scale study uses multiple intelligence theory, systems theory, and 
psychometric theory as the theoretical framework for data collection and analysis. Finally, the IPA scale study uses 
all four theories. The goal is to produce an integrated performance assessment methodology to assist organizations 
and professionals with selection, placement, career planning, and professional development activities. 

Conclusion 

Before energy could be expended to create methods and tools to enable practitioners to measure and track 
individual performance against industry competency standards a literature review was conducted to determine what 
competencies, methods, and tools were extant in the field. While there are clearly defined competency standards and 
several types of methods available to guide performance measurements, there still remain no tools readily available 
that practitioners can use to measure performance in current, new, or emerging competency domains nor is there an 
integrated performance assessment methodology available to enable holistic measurements of individuals from 
multiple perspectives across all competency domains. For example, IT professionals are the architects of multiple 
types of instructional solutions and often develop and construct learning experiences designed to cultivate and 
transfer knowledge (Tracey & Richey, 2007; Tracey, 2001). The skills IT professionals need to maintain multiple 
roles also require them to possess competencies in more than one content domain. “Playing [with multiple] roles is 
analogous to maintaining a collection of hats - when the situation calls for it, the professional slips out of one role 
and ‘puts on’ another” (Bernthal et al., 2004, p. xxiii). IT professionals must produce learning experiences and 
create environments that capitalize on the learning styles of learners while providing opportunities for learners to 
develop their multiple intelligences. Similarly, in order to be effective in maintaining multiple roles, IT professionals 
must also develop the same multiple intelligences that they seek to impart to the learner. 

As noted earlier, previous studies of ISD competencies have not looked at or explored ways in which IT 
professionals could be measured on ISD competency standards (see NWCET, 2003; Richey et al., 2001; Atchison, 
1990; Song, 1998). Second, there has been only one attempt to develop a measurement to assist IT professionals 
with assessing their skills on ISD competency standards (see Stepp, 1990). Finally, seminal studies undertaken to 
describe ISD practice prior to studies to define and validate ISD competencies were primarily qualitative. 

Ultimately, the IPA research methodology stands in stark contrast to these previous qualitative studies because the 
long-term goal of this research methodology is to establish an objective, valid, and reliable systemic measurement 
process to assist IT professionals and organizations with assessing skill capabilities using quantitative and 
qualitative methods. The ISD Performance Inventory study, is the first study implemented, and has the short-term 
goal to establish a valid and reliable measurement instrument to assess the skill capabilities of IT professionals. The 
results from this study will lead to studies two, three, and four. A unique skill capabilities profile outlining a 
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professional’s strengths and weaknesses provides guidance in skill areas where he or she may need improvement. 
Professionals could then use their individual profdes as a way to self-regulate their performance as a part of career 
and professional development activities. Employers could use the profile to make more effective career planning 
decisions regarding existing employees and make better hiring decisions regarding new job candidates. Educational 
organizations could also use the profile to provide direction and guidance about training and academic program 
offerings. 
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Changing the Kaleidoscope for End-of-Course Surveys 

Michael Chronister and David Pedersen 



End-of-course surveys are an important and often controversial part of the Kaleidoscope of academic 
performance assessment. For that reason changes to the process at Embiy Riddle Aeronautical University were 
approached cautiously. After years of discussion, an initiative was begun to migrate from paper-based end-of- 
course surveys to online surveys. Choosing the appropriate vendor was a challenging, but crucial step in the 
implementation process. Implementation began with a small pilot study that compared paper and online results. 
Based on the success of the pilot study, full implementation was initiated. At that point several challenges emerged 
that required adaptation and resulted in some lessons learned that should be instructive to others considering the 
move to online surveys. 



The Initial Condition 

Embry-Riddle Aeronautical University is a multi-campus university with two traditional residential 
campuses, 130 teaching centers world wide, and a distance learning program. End-of-course evaluations were 
conducted differently at each campus. Most were conducted using a traditional paper-based instrument, but 
questions varied from campus to campus, and the distance learning program used a home-grown database to collect 
course evaluation responses. 

After years of discussion, initiatives from all three campuses intersected in the summer of 2004 with 
independent requests from each campus for the Information Technology department to implement online surveys. 
One campus was in a beta project to develop a new open source application for online surveys. Another campus 
was focused on the broader question of evaluating faculty performance using a variety of instruments including 
online surveys. The third campus had a faculty senate sub-committee looking at online survey vendors. The efforts 
of the three campuses finally merged with a meeting of campus representatives convened by the Provost in the 
spring of 2005 that resulted in a university-wide selection initiative. 

The Selection Process 

Following the 2005 summer break, a committee was appointed by the provost’s office consisting of campus 
representatives from the faculty, teaching and learning centers, information technology, institutional research, and 
the assistant provost. During the summer months prior to establishing the committee, the educational technologist 
who had been working with the three campuses conducted a review of the five vendors under consideration 
regarding critical elements of the online survey process. A report was created that compared the following 
requirements and also included information concerning each company’s experience, customer base, and any unique 
characteristics. 

• Flexible Survey Authoring - The ability to create multi-tiered surveys that can include questions from the 
perspective of the instructor, the department, the college, the campus, and the university. 

• Controlled Distribution - The ability to distribute and submit surveys via the university portal. Students 
would be authenticated to complete surveys only for the courses in which they are enrolled. Authentication 
would require an interface with the student data system to provide user authentication and course 
enrollment data. 

• Data Collection and Analysis - The ability to track responses and promote participation. Responses are 
collated and analyzed appropriately for the instructor, department, college, campus, and university levels. 
Data is available for further analysis. 

• Tailored Reporting - Data are reported in appropriate information formats for varied users. Instructors 
would receive information relevant only to their courses, while departments, colleges, and campuses would 
receive summary data appropriate to their needs. 

The overall result of the reviews indicated that online surveying is a relatively new endeavor, all of the 
solutions lacked robustness of features, and vendors were in the process of improving their offerings by adding 
functionality or creating new solutions. 
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Using the report, the committee asked three vendors to provide demonstrations of their products and 
narrowed the field to a single selection. At that point, the Institutional Research Department requested that 
consideration be given to the vendor they were currently using to conduct general surveys. Because that department 
would be managing course surveys, and their vendor had just announced an integration component for the 
university’s course management system, a demonstration of the product was arranged. Results indicated that the 
extensive customization needed to make their solution viable would be untimely and too costly. 

Implementation of the System 

The selection committee’s consensus recommendation was forwarded to the administration for approval in 
April 2006. Due to funding and acceptance considerations, it was decided to conduct a pilot test of the system for 
the fall term. The pilot was conducted with 2 1 instructors who taught two sections of the same course. One section 
submitted surveys online; the other section submitted paper-based surveys in the classroom. The return rates for the 
online administration were about 30% lower than for class room administration. However, the online respondents 
provided 15% more comments than the paper-based respondents. Comparison of the paper-based vs. online results 
found statistical differences in only 3 of the 1 1 items at just one of the campuses with online results being slightly 
less positive. While there were some concerns with return rates and slightly less positive responses, the committee 
recommended proceeding with full implementation. 

Full implementation of the online course evaluation system began in spring 2007. ERAU faced challenges 
as would be expected with the rollout of any new process or system. The majority of the problems encountered in 
the implementation of the online evaluation system can be traced to three main sources: inconsistencies in the data 
from the student information system, problems with the vendor, and ineffective internal communication. 

Data in the Student Information System: 

As the distribution of the evaluations and the aggregation of the results are based on course enrollments, the 
most important consideration in the implementation of an online course evaluation system is the accuracy of the data 
in the student information system. Embry-Riddle IT created a system that captures data from the SIS and 
automatically formats it to be uploaded into the online evaluation system according the vendor’s specifications. IR 
and IT met with individuals that were most familiar with the student information system and were assured that the 
data we planned to use was accurate and current. This proved true in better than 99% of the cases; however, some 
inconsistencies were found in key fields such as faculty department, course end date, and user ID. These rare 
inconsistencies caused errors in the automated process and lead to a great deal of manual manipulation of the data. 

Lesson Learned: A more thorough accounting of all data in all key fields as well as additional flexibility in 
the automated process would have ensured an easier integration of the online evaluation system. 

Verify the Vendor’s Work: 

The vendor created a customized report database to meet the specific reporting requirements of Embry- 
Riddle. However, the database was not delivered at the time of the pilot test. As there would be several months 
between the pilot test and full implementation of the system, ERAU accepted on good faith the vendor’s promise 
that the report tool would be made available for the spring 2007 implementation and agreed to a contract. However, 
the report database was delayed beyond the implementation and reports were not available to instructors or 
administrators until several months after the initial evaluations were completed. This delay caused frustration and 
dissatisfaction among the faculty. 

One of the key requirements of the system was that users would be authenticated and have access to the 
online evaluations via the university portal. This was accomplished using token authentication between ERAU and 
the vendor. ERAU setup to deliver the token as described in the Software Requirements Specifications provided by 
the vendor. Unfortunately, in some instances the vendor did not follow their own specifications. As a result, the 
authentication worked during business hours, but for several hours each night the tokens did not match, leaving the 
evaluations inaccessible. This periodic lack of availability was obviously problematic for our students and faculty in 
other time zones and hurt the response rates and the general credibility of the system. ERAU IT implemented a 
testing protocol to automatically test the authentication system periodically and log any failures. This facilitated 
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diagnosis of the problem and will ensure that the administrators will be aware of any future outages. 
Implementation of such a testing procedure at the outset of the system would have picked up on the discrepancy in 
the documentation. 



Lesson Learned: Verification of all of the deliverables and establishment of protocols to test the vendor’s 
work would have ensured that the system was fully functional prior to full implementation. 

Communication: 

Effective communication within any large organization is difficult, and with its international teaching sites 
and distance learning programs ERAU is no exception. In addition as the online evaluation system required a 
completely new process, a great deal of information had to be made available to a large number of people. Details 
regarding when the evaluation would be available, how users would be notified, instructions for students to access 
the evaluations, who has the ability to view the results, and how to access the reports needed to be communicated. 
The administrators of the online evaluation system notified the university community of the new system and 
provided training materials for users well in advance of the implementation. In addition, the section of the 
university policy and procedure manual related to course evaluations was revised. All such communications were 
conducted by email as it is the official outlet for university announcements. However, it became apparent email is a 
less than reliable form of communication. Email is easily blocked, lost, deleted or never delivered. In addition, 
some students, faculty, and staff do not check their university email accounts, and if they do, they do not thoroughly 
read long, detailed messages or updated policy manuals especially if some of the information does not apply directly 
to them. The difficulty in providing each user with the information they needed and nothing more created confusion 
and dissatisfaction with the online evaluation process. 

Lesson Learned: The administrators of the online evaluation system have created an informational 
webpage with all of the training materials and policies available to the users. Instead of pushing the entirety of the 
information to all of the users, they can direct the users to this clearinghouse where they can find the information 
and support they need. 

Reflection on the pilot test: 

The pilot test was primarily conducted in order to determine if the online evaluations would achieve 
acceptable response rates in order to reassure the faculty that the change in the method of evaluation would not 
negatively impact the results. The pilot proved quite valuable in this effort. However, the small number of courses 
chosen for the pilot did not provide a sufficiently large sample to effectively evaluate the system for errors and 
inconsistencies. Many of the difficulties noted above would have been caught by a larger pilot test, allowing them 
to be addressed prior to full implementation. 



Overall Lessons Learned 

1 . Including appropriate representation from each campus in the selection process was critical. 

2. Centralizing the selection process from the Provost’s office at the outset would have cut the selection time 
considerably. 

3. Coordinating the selection process with the academic calendar would have made the process more efficient. 

4. The pilot provided valuable supporting evidence for the viability of full implementation, but it failed to 
reveal some of the problems encountered in full implementation. Increasing the size of the pilot would 
have been helpful. 

5. Technical solutions must function properly. Any failure causes dissatisfaction for the whole application. It 
is important to adequately test all systems prior to full-scale implementation. 

6. Effective communications to constituents is critical. Multiple methods should be used to disseminate 
information. 
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Summary 



Changes to the faculty evaluation process are highly charged events. Migrating end-of-course surveys from 
paper to online was a courageous first step in a broader initiative of implementing technology to make assessment of 
academic performance more efficient and effective. It is anticipated that future endeavors will include electronic 
portfolios for faculty performance evaluation and program assessment, and discussions have begun on implementing 
an institutional assessment package that is now part of the university’s course management system. Experience 
gained from implementing online surveys should inform the process of implementing additional technologies that 
will continue to change the assessment Kaleidoscope at Embry-Riddle Aeronautical University. 
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Managing Knowledge Work 



Mark Salisbury 
University of New Mexico 



This article describes the iLearning framework for managing knowledge work in organizations. This 
framework was initially used to successfully build a knowledge dissemination system for the laboratories and 
facilities that are under the direction of the United States Department of Energy (DOE) (Salisbury & Plass, 2001). 
The follow-on work to this effort was the development of a collaboration application that fed the dissemination 
system for the DOE laboratories and facilities. The resulting system managed the life cycle (creation, preservation, 
dissemination and application) of knowledge for the DOE laboratories and facilities (Salisbury, 2003). Recent 
work has focused on extending the theoretical foundation of the framework to improve collaboration and in methods 
to identify performance objectives of knowledge work for reusing and repurposing that work. In the next section, 
the life cycle of knowledge in an organization is discussed. Next, the theoretical foundation for the iLearning 
framework is described. Afterwards, Technologies used to manage the life cycle of knowledge in an organization 
are highlighted. Finally, the other aspects of the framework — Work Processes, Learning Processes, and 
Methodologies — that support Technologies for integrating knowledge into a collaborative work environment in an 
organization are discussed. 



The Life Cycle of Knowledge in an Organization 

Successful organizations have learned to manage the ongoing life cycle of knowledge - its creation, 
preservation, dissemination, and application. The first phase, creation of new knowledge, takes place when 
members in the organization solve a new unique problem, or when they solve smaller parts of a larger problem such 
as the ones generated by an ongoing project. The next phase is the preservation of this newly created knowledge. 
This includes recording the description of the problem as well as its new solution. This phase feeds the next one, the 
dissemination and application of this new knowledge. The dissemination and application phase involves sharing 
this new knowledge with the other members of the organization. It also includes sharing the solutions with the 
stakeholders affected by the problems that were solved. Disseminated knowledge then becomes an input for solving 
new problems in the next knowledge creation phase. An organization’s ability to solve problems increases with the 
utilization of this disseminated knowledge. In this way, each knowledge life cycle phase provides input for the next 
phrase - creating an ongoing cycle. Since this cycle continues to build upon itself, it becomes a knowledge spiral in 
the organization as described by Nonaka and Takeuchi (1995). 

While the growth and sharing of knowledge is recognized as one of the most important elements in 
becoming a learning organization (Easterby-Smith, 1997; Marsick & Watkins, 1994; Senge, 1990), what has been 
missing, according to many researchers and practitioners in the field, is the development of a theoretical foundation 
for describing how people learn and perform in an organization (Raybould, 1995; Salisbury, 2000). This theoretical 
foundation is needed by today’s organizations to avoid the development of technological solutions that do not 
support their entire life cycle of knowledge (Plass & Salisbury, 2002). To address this situation, a theoretical 
foundation for integrating learning into collaborative work was developed. It describes how learning can take place 
with one individual, be preserved, and transferred to other individuals in an organizational setting (Salisbury & 

Plass, 2001; Salisbury, 2003). 



Theoretical Foundation for iLearning 

To represent the complexity of organizational knowledge, a revision of Bloom's Taxonomy (Bloom, 1956) 
developed by Anderson, Krathwohl, Airasian, Cmikshank, Mayer, Pintrich, Raths & Wittrock (1998) was used to 
provide the basis for extending the description of knowledge utilized within the iLearning framework. One of the 
major differences in the revised taxonomy by Anderson et. al. (1998) is the identification of knowledge as a separate 
dimension that describes it as factual, conceptual, procedural, and metacognitive. Another major difference is that 
Anderson and colleagues recast Bloom's other categories into a "process dimension" which describes the learner’s 
cognitive processes when processing knowledge of that category. These process dimension categories were also 
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renamed from Bloom's original "knowledge, comprehension, application, analysis, synthesis, and evaluation" to 
"remember, understand, apply, analyze, evaluate, and create." Note that Anderson and colleagues place "create" as 
the highest level of cognition; it describes individuals putting elements together to form a novel coherent whole or 
make an original product. 

Anderson et al. (1998) describe factual knowledge as terminology, specific details, and elements. 
Conceptual knowledge relates to theories, models, principles, and generalizations. Procedural knowledge includes 
skills, algorithms, techniques, and other methods that are specific to a product or process. Metacognitive knowledge 
was added by Anderson and colleagues to Bloom's Taxonomy. It is "knowledge about knowledge" and involves 
general strategies for learning, thinking, and problem solving. Metacognitive knowledge also includes knowledge 
concerning the appropriate contexts and conditions for the use of the strategies themselves. Additionally, it includes 
the “heuristics” or “rules of thumb” that experts use to solve problems. 

At the individual level, the theoretical foundation has elements of Situated Cognition as described by 
Brown, Collins & Duguid (1989). The theoretical foundation supports learning in the context of the work at the 
moment - creating an “authentic context” for learning. Knowledge workers can access knowledge - and other 
people — to learn how to construct solutions to pressing organizational problems in a just-in-time manner. 
Furthermore, the theoretical foundation supports Situation Cognition for learners with differing cognitive needs by 
providing different types of knowledge as defined by Anderson and his colleagues (1998) in their revision to 
Bloom's Taxonomy (factual, conceptual, procedural, and metacognitive). As a result, the theoretical foundation 
supports work and learning to “live in the same space, “occur at the same time,” and become interdependent. As a 
result, learning is situated in the authentic task of organizational work and takes place during that work. 

At the team level, the theoretical foundation is an extension of the theory of distributed cognition (see 
Salomon, 1996, for an overview of distributed cognition). One of the best documented examples of distribution 
cognition in a work environment is by Edwin Hutchins in his book “Cognition in the Wild” (Hutchins, 1996). 
Hutchins studied how a crew collaborated to operate a large ship at sea. According to his description of the theory 
of distributed cognition, cognition is distributed across individuals. That is, no one individual has complete 
knowledge as to how to accomplish a complex task such as operating a large ship. Hutchins also describes that 
cognition is distributed across the artifacts of an organization’s work. On the ship that means the instruments 
provide critical decision-making information to the crew members. And, according to the theory of distributed 
cognition, cognition is in the history of those artifacts. On the ship, the previous version of an instrument gives a 
context for the present version of that instrument. In an office environment, artifacts are the knowledge products of 
the organization. These are the “intermediate products” of a larger process and are such things as design documents 
and quality plans. Another set of artifacts are the knowledge assets that document the organization’s processes, 
instruction, work examples, and expert advice that are used as resources by the members of the organization to make 
the knowledge products. In the theoretical foundation, the theory of distributed cognition is extended to involve 
different types of knowledge as defined by Anderson and his colleagues (1998) in their revision to Bloom's 
Taxonomy (factual, conceptual, procedural, and metacognitive); these different types of knowledge are present in 
the distribution of cognition across individuals, their artifacts, and the history of their artifacts. 

At the organizational level, the theoretical foundation is an extension of Nonaka and Takeuchi’s (1995) 
description of creating a knowledge spiral in an organization. In Nonaka and Takeuchi’s knowledge creation 
process, transferring knowledge from one organizational member to another begins by the first member converting 
tacit knowledge (intuitions, unarticulated mental models and embodied technical skills) into explicit knowledge (a 
meaningful set of information articulated in clear language including numbers or diagrams). This explicit 
knowledge can then be passed on to another member of the organization — who must convert it into tacit knowledge 
(internalization) before he or she may use it. Again, the theoretical foundation extends this description of 
knowledge creation by identifying the different categories of knowledge as defined by Anderson and his colleagues 
(1998) -- factual, conceptual, procedural, and metacognitive — that are involved in the knowledge creation and 
transfer process. 
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Technologies 



In the following example, disseminating and applying knowledge begins with an engineer that needs to 
make a quality plan for a new product. The engineer goes to the system (perhaps, from a cell phone), clicks on the 
area of Design, then clicks on the area of “Detailed (Design),” and drills down to the area of “Quality Plan.” There 
the engineer finds all the materials that he or she will need to develop a Quality Plan. There will be documents 
describing what needs to be addressed in the Quality Plan. There will be instruction available on the general 
principles and techniques behind a Quality Plan. The instruction addresses the “why” part — that is, why do we 
need a quality plan? There are also examples available of successful quality plans. They illustrate how someone 
applied the general principles of developing Quality Plan to a specific project. Finally, there is expert advice 
available that provides some direction as to when to use one approach over another when developing a Quality Plan. 

But that’s not all the engineer would find at that Quality Plan area in the system. The engineer would also 
find links to the people that are responsible for the content of the area. There is contact information for the creators 
of the documents, instruction, examples, and expert advice. In contacting these content providers directly, the 
engineer has the opportunity to understand the subtleties of the content and its application to specific projects. Note 
that with these resources - materials and an opportunity for an exchange with the people who created them - the 
engineer can learn what is needed to get the job done. In this case, it’s the creation of a Quality Plan. With adequate 
materials and the help of others, the engineer learns - only what is needed, in a “just in time” manner — to create the 
Quality Plan. And it could be done with a portable device such as a cell phone. This is learning situated in the 
context of an authentic task - the pressing work of the moment. It describes the essence of Situated Cognition as 
described by Brown, Collins, and Duguid (1989). 

However, when the engineer creates the Quality Plan - it is just the first step in completing a finished 
Quality Plan in a collaborative work environment. The next step is a review step - followed by an approval step. 
Note that all the assets that were available to the engineer to create the Quality Plan - materials and an opportunity 
for an exchange with the people who created them - are available to the other people involved in the review and 
approval steps. They, along with the engineer that created the Quality Plan, have integrated learning as they work 
together in a collaborative work environment. 

Integrating Learning into Collaborative Work 

This example described above is the result of integrating learning into a collaborative work environment for 
an organization. However, what we have seen is simply the technology that serves up the information. Technology- 
based solutions leave us here - wondering how the information gets into the system - and more importantly, how is 
it updated and maintained. It’s quickly apparent that the technology is simply the “tip of the iceberg” - a byproduct 
of integrating learning into collaborative work — that has provided this information. As Figure 1 shows, this article 
discusses the foundations, processes, and methodologies needed to support the technologies for integrating learning 
into a collaborative work environment. 

Take, for example, the business process for a manufacturing company with the two main steps of “Design” 
and “Build” for the manufacturing process. The Design step contains two sub-steps - “Preliminary” and “Detailed.” 
The Build step also contains two sub-steps - “Implementation” and “Delivery.” According to the theory of 
distributed cognition (Solomon, 1996, Hutchins, 1996), all the subtleness of a complex process does not reside in the 
head of one individual. While each member of the organization knows how to do his or her part of the process, the 
larger process is known only collectively - the ability to make informed decisions within the process is distributed 
across all people who work the process. 
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Figure 1 . iLearning Framework for Managing Knowledge Work 



Work Processes 

This example also illustrates the second aspect of the theory of distributed cognition (Solomon, 1996, 
Flutchins, 1996). That is, cognition is distributed in the artifacts of the workflow process. Artifacts are used to 
capture decisions and information about the work that has been done in the workflow process The Design 
Document, the Quality Plan, the Testing Report, and the User Document are the artifacts for the example workflow 
process. Since they each have embedded knowledge about decisions that concern a unique aspect of the process, 
they each also represent a subset of the cognition needed to complete the entire workflow process. (See Nemeth, 
Cook, O’Connor, & Klock, 2004 for an overview on the importance of cognitive artifacts to the theory of distributed 
cognition.). 

Additionally, this example shows the third aspect of the theory of distributed cognition — the history of an 
artifact reveals the context for decisions and information about the process over time (Solomon, 1996, Flutchins, 
1996). For example, if the Quality Plan is currently in version 2.0. This means that there were some major changes 
in the Quality Plan since version 1.0. The history of changes in an artifact tells the reasons “why” those changes 
were made. Frequently, it turns out that artifacts are historically related to one another. For example, when the 
Design Document goes from version 1.0 to 2.0, the Quality Plan will also go from version 1.0 to version 2.0 since 
the Quality Plan is dependent on the Design Document. In this way, the histories of artifacts provide important 
reasoning about their present form. 

Up until this point in the article, the product of work was referred to as an “artifact.” Recognizing the 
embedded nature of knowledge in these artifacts, they are referred to as “knowledge products” throughout the rest of 
this article. Every knowledge product has a set of criteria, or performance objectives that need to be met by its 
developers for its successful completion. These performance objectives are sometimes implicit - or in the “eye of 
the beholder.” Recognizing the existence of these performance objectives but not able to easily articulate them is 
found in such phrases as “1 know a good Quality Plan when I see one” or “shouldn’t a Quality Plan have a. . .” 
Uncovering these underlying performance objectives is essential for improving the quality of the ongoing work in an 
organization. These performance objectives tell an organization what needs to be done and how well it should be 
done. 
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Borrowing from the field of Instructional Systems Design (ISD), one way to go about identifying 
performance objectives is to conduct a content analysis. A content analysis always starts off with the same question, 
“What knowledge does a person need to know to create this knowledge product?” (Davis, Alexander, and Yelon, 
1974). That is, it focuses on identifying the cognitive skills needed to create the knowledge product. Cognitive 
skills underlie learning how to learn, that is, getting at the heart of the problem (Gagne, Briggs, and Wager, 1992). 
Once the knowledge is identified, it is listed by topic and each topic is rewritten as a performance objective. For 
example, the topic “Completeness and Correctness Criteria” is rewritten as the performance objective “In the 
Quality Plan, the developer will list all approved criteria forjudging the plan as complete and correct.” (For a 
complete description of the steps for conducting a content analysis and an overview of the ISD process, see 
Rothwell & Kazanas, 2004). The process of stating performance objectives begins by identifying the kinds of 
objectives that must be written. The most commonly used classification system for performance objectives was first 
described in 1956 by Bloom and his colleagues (Bloom, 1956). Performance objectives make a precise statement of 
what learner should “do” in order to accomplish the stated performance (Mager, 1997). They contain a performance 
component, a criterion component, and a condition component. The performance component describes how 
proficiency will be demonstrated. The criterion component describes how well the proficiency must be preformed. 
And, the condition component describes what conditions must exist when the proficiency is demonstrated. 
Performance objectives of a knowledge product provide the basis for creating metrics to measure the knowledge 
work of organizations. Measuring how well the performance objectives have been met provides data relating to the 
“quality” of the knowledge product. Measuring how much time is spent in creating a knowledge product provides 
data relating to scheduling and cost for the knowledge product. 

Learning Processes 

Figure 2 shows that when Anderson and colleagues revised Bloom's taxonomy, they made knowledge a 
separate dimension with four categories: factual, conceptual, procedural, and metacognitive (Anderson, et.al., 1998). 
They recast Blooms’ other categories into a “process dimension” which describes the learner’s cognitive processes 
when solving a problem in that category. Figure 2 also shows that novices are usually working at the level of trying 
to understand and remember. This is why it takes novices so long to get anything done. They are really “stuck” at 
the level of just trying to “get what’s going on” and put it to memory. Also, Figure 2 shows that practitioners are 
usually working at the level of analyzing the situation and applying knowledge to form a solution. They already 
understand what to do and remember how to do it. Give them a problem similar to one that they have solved before 
and they will quickly analyze the problem and take a previous solution, adapt it, and apply it to their new problem. 
Finally, Figure 2 shows that experts should be working at the level of evaluating solutions and creating new and 
unique ones. The word “should” is put in this explanation because if an organization is using its experts like 
practitioners - doing the everyday work - then the organization is not getting the most from its experts. If the 
organization’s experts are spending all their time on the work of the day, then the opportunity is lost for better ways 
to do tomorrow’s work. 

Figure 2 illustrates how to provide learners with appropriate knowledge assets. Of course, an appropriate 
knowledge asset depends on the type of knowledge that they seek. Novices use the system to become practitioners, 
practitioners use the system to become experts, and experts utilize the system to create new knowledge. In the 
process of becoming practitioners, novices seek to understand and remember conceptual knowledge. Instructional 
materials are appropriate knowledge assets for them as they provide access to conceptual knowledge. Note that 
novices will still require factual knowledge to hilly understand and remember the conceptual knowledge — similar to 
a student requiring access to the manual to understand the instruction presented in the classroom. In the process of 
becoming experts, practitioners utilize examples to analyze and apply procedural knowledge. Note that practitioners 
will still require factual and conceptual knowledge to apply and analyze procedural knowledge. Experts create and 
evaluate expert advice. By doing so, they provide access to metacognitve knowledge for others in the organization. 



293 




Experts • 



Practi- 

* 

tioners 



Novices • 



Figure 2. Differentiating Learners and the Knowledge They Seek 

Methodologies 

Considerable attention has gone into developing methodologies for reusing knowledge work in recent 
years. Much of it has focused on the methodologies for developing “learning objects” or “content objects” (Barritt 
and Alderman, 2004; Hamel and Ryan- Jones, 2002; Rehak, 2003; Robson, 2002). However, while quite a bit has 
been published on sharing knowledge, especially, in the area of communities of practice (Brown and Duguid, 200 1 ; 
Lave and Wenger, 1991), little has focused on the mechanics of how to identify and track knowledge for reuse 
(Osterlund and Carlile, 2005; Wiley, 2004). The result has been that for most organizations, reuse is addressed only 
at the institutional level, if at all (Davenport, 2004). 

The following example shows how performance objectives can be utilized for reusing knowledge work. It 
begins with two performance objectives that were originally developed for different tasks (writing Quality Plans and 
Testing Reports) and described differently — but were later found to be fundamentally the same. This created the 
opportunity for reusing a knowledge asset. Since both performance objectives could now have the same identical 
text, this text can be a single document that is referenced by both performance objectives. This way, whenever the 
document for this combined performance objective is changed, it will be changed for users no matter which 
knowledge product they are working on (Quality Plan or Testing Report). 

Figure 3 shows how performance objectives can be utilized for repurposing knowledge work. Performance 
Objectives 3, 6, 7, and 8 are very similar; workers will apply the same general principles and techniques to satisfy 
them. That means that a “common” document that describes what needs to be done can be used for all four 
performance objectives. This is also true for “common” instruction that describes why things need to be done and 
some “common” expert advice that describes how to do it. The document, instruction, and expert advice are shared 
knowledge assets. However, Figure 3 shows that not all knowledge assets are shared between the four performance 
objectives. Each performance objective has its own unique set of knowledge assets that describe the context (place 
in the process, physical site) in which the performance objective is addressed. This situation forms the basis for 
repurposing a knowledge asset. For example, when the Performance Objective 3+6+7+8 is accessed by workers 
writing a Quality Plan from Site A or Site B - or workers writing a Testing Report from Site C or Site D - they will 
see a common document, instructional module, and expert advice. (The “+” operator means that the four 
performance objectives share common knowledge assets, but each has additional knowledge assets that are not 
shared with the other others.) However, depending on what part of the process they are coming from (Quality Plan 
or Testing Report) or what site they are coming from (Site A, Site B, Site C, Site D), workers will see additional and 
different “contextual” knowledge assets. That means a worker from Site A trying to write a Quality Plan would see 
a document, an example, and expert advice specifically tailored for Site A, while a worker from Site B trying to 
write a Quality Plan would see a document, an example, and expert advice specifically tailored for Site B. On the 
other hand, a worker from Site C trying to write a Testing Report would see document, an example, and expert 
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advice specifically tailored for Site C. And to be complete, a worker from Site D trying to write a Testing Report 
would see document, an example, and expert advice specifically tailored for Site D. 



SiteA 



Creating 

Completeness and 
Correctness Criteria 



Hite H 



SiteC 



SiteD 




For All Knowledge 



Products 



Applying 
Completeness and 
Correctness Criteria 



Expert Advice 
Instruction 
Document 



Knowledge 

- 

Products 

Performance 

>- - 

Objectives 



SiteA 



QualityPlan 



Site B 



SiteC TestingReport 











-»| Expert Advice 


-►1 Expert Advice 


-*| Expert Advice 


-♦) Expert Advice 


■*| Example 


-») Example | 


-*\ Example | 


-*| Example j- 


Document 


-*| Document 


Document 


~*| Document 



For Process and Site Specific Knowledge Products 

Figure 3. Repurposing Knowledge Assets 



Discussion 

This article describes the iLearning framework for managing knowledge work in organizations. The 
theoretical foundation for the iLearning framework, details how learning can be supported at the individual, team, 
and organizational levels. At the individual level, the iLearning framework supports learning in the context of the 
work at the moment - creating an “authentic context” for learning. At the team level, the iLearning framework 
supports learning in the context of a “distributed environment” where cognition is distributed across individuals, 
their artifacts, and the history of their artifacts. And, at the organizational level, the iLearning framework supports 
creating a knowledge spiral in an organization where transferring knowledge from one organizational member to 
another begins by the first member converting tacit knowledge into explicit knowledge before passing it on to 
another member of the organization — who must convert it into tacit knowledge before he or she may use it. The 
iLearning framework also supports “different types” of learning at the individual, team, and organizational levels. It 
supports novices, practitioners, and experts in their need of different types of knowledge: factual, conceptual, 
procedural, and metacognitive. 

Another aspect of the framework presented in this article is the use of performance objectives to uncover 
the drivers of knowledge work. These performance objectives provide the key for improving the workflow process 
and overall knowledge worker productivity. They determine what to measure for providing feedback and how to go 
about making improvements in the knowledge work. They also determine what knowledge to reuse and repurpose 
and why it should be reused and repurposed. 

The major aspect of the iLearning framework, however, is the emphasis on improving learning in a 
continuous and integrated way in organizations. In the last twenty years, there has been a lot of thought and work in 
the area of managing work processes. Organizations have recognized the value for examining how they do their 
work and how best to optimize their processes to get the work done better and faster. At the same time 
organizations have recognized the value that learning as a group - or organization - improves the organization’s 
ability to get the work done under changing circumstances. While a lot has been written about the learning that 
goes on in organizations, little attention has been placed on modeling that learning - at least, not in the same vigor 
that has been done with modeling and optimizing work processes. This article has put forth a means to model the 
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“learning processes” of an organization. The premise is that for organizations to reach their potential, they must 
integrate learning into their work. Or said another way, effective organizations must be able to work and learn 
together — concurrently. That means that the “learning processes” must be modeled and combined with the work 
processes. That’s why the “pyramid” model for this article (Figure 1) has the work processes and learning processes 
on the same layer - it shows they “live in the same space, “occur at the same time,” and are interdependent. 

The view of learning presented in this article is an entirely different view of learning than the one based 
upon the “learning occurs after training” approach. In the learning occurs after training approach, training is done 
for tomorrow’s production. When training is complete, workers will be able to apply that training when the 
opportunity presents itself. As a result of this view, training is typically looked upon as a “non-critical” input to 
production. It can be delayed, or eliminated, because there is enough time to develop a work-a-around for the 
missed training before it can affect tomorrow’s production. 

In a contrary view presented in this article, learning is part of the work process in the “learning during 
work” approach, and it has to occur during today’s work process to get today’s work done. It is essential to today’s 
production, and without it, the work does not get done right and on time. In this view, eliminating learning, or 
delaying it, only reduces an organization’s ability to get today’s work done. Consequently, learning is looked upon 
as a critical part of the work process. 

Methodologies are built upon the work and learning processes. That is, methodologies are used to model 
the products of knowledge work, the performance objectives of knowledge work, and the knowledge assets that are 
applied to complete the work. These methodologies define the granularity of knowledge assets, how they will be 
created, stored, displayed, and updated. Finally, readers are shown that managing knowledge assets by the 
performance objectives they address is the key to the reuse and repurposing of those assets. 

Future Directions 

Further work is needed to develop the interventions necessary to realize the whole of the pyramid for 
managing the life cycle of knowledge in an organization. One area should focus on empowering a leader to create a 
vision for managing the life cycle of knowledge in his or her organization. This vision includes why the “whole 
brainpower” of an organization is greater than the sum of its parts. It shows that value lies in the knowledge 
provided to customers and the only way to increase that value is by bringing more brainpower to it. And most 
importantly, it helps organizational members to paint a picture of what managing the life cycle of knowledge will 
look like in their organization. 

Further work is also needed to utilize performance objectives for evaluating the performance of knowledge 
workers. From this perspective, performance should be evaluated in terms of the knowledge that individuals bring 
to bear on the problems of the organization. The contribution of individuals to the organization’s “stockpile” of 
factual, conceptual, procedural, and metacognitive knowledge can be used as an information source for individual 
performance assessments. Obviously, a “count” could be conducted to quantify contributions to procedure manuals, 
online instructional modules, documented work examples, and recorded expert advice. However, as discussed in the 
section on learning processes, these contributions can take place informally — sharing a fact, providing on the job 
instruction, sharing an example, or giving a nugget of expert advice. Further research is needed into the 
development of new methods for using performance objectives to “measure” and track these contributions. 
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Student Response to an ePortfolio Initiative: 

A Grounded Theory Analysis 

Thelma Seyferth, Albert D. Ritzhaupt, Oma B. Singh, Robert Dedrick, Ph.D. 

University of South Florida 

Abstract 

With the proliferation of eportfolios and their various uses in higher education, it is 
important for educators and other relevant stakeholders to understand the contribution of the 
student perspective to the planning and implementation of eportfolio initiatives. This research 
describes an eportfolio initiative at a large public university in the southeastern US. Data on 
student perspectives were analyzed using a grounded theory approach. Three interrelated themes 
emerged from the qualitative analysis of these data: purpose, support, and personal impact. The 
success of the eportfolio initiative is discussed in terms of these three themes, using the 
ePortfolio Implementation Support Model (ePISM). Results suggest that data on student 
perspectives are valuable not only in terms of evaluating the success of the initiative in reaching 
learner-centered goals, but also to inform the design and ongoing improvement of appropriate 
support. 



Keywords: eportfolios, student perspectives, grounded theory. 

Portfolios to ePortfolios in Education 

The emphasis on accountability in the field of education brought on by the No Child Left Behind Act of 
2001 (U.S. Department of Education, 2002) has increased the need for authentic forms of assessment. As educators 
have begun to measure learning based on performance on real-world tasks, it has become clear that a method of 
documentation is necessary to support this process. From the fields of art and journalism educators have borrowed 
the concept of portfolio assessment to meet this need. Fueled by the pedagogical pendulum swing toward 
constructivism, whereby students are given the tools for learning (Jacobsen & Spiro, 1995) and are guided toward 
self-discovery (Meeus, Questier, & Derks, 2006; Strudler & Wetzel, 2005), the portfolio method has seen 
widespread adoption in education. Salzman, Denner, and Harris (2002) report that 89% of colleges or departments 
of education are using portfolios as one form of assessing student outcomes. 

Educational portfolios, constructed over a period of time, are a purposeful collection of work that can be 
used to document accomplishments, growth in knowledge, competencies, and skills, or provide a snapshot of 
learning at a given point in time (Barrett, 2001; Lorenzo & lttelson, 2005). Wheeler (2003) suggests that the 
defining characteristic of a portfolio is its purpose. Similarly, Barrett (2007) notes that “the term portfolio should 
always have a modifier or adjective that describes its purpose.” The intended purpose of the portfolio forms the 
basis for the inclusion of content. The versatility of portfolios has made them popular devices for a variety of 
purposes in education. Two basic purposes emerge from the literature: assessment and representation. For purposes 
of this research, we will consider only the assessment portfolio. 

Portfolios can support both assessment for learning and assessment of learning. An assessment for learning 
portfolio is used to document progress in learning over a period of time. This type of portfolio supports the reflective 
practice that is essential to continuous improvement. The content of such a student-centered learning device is 
generally negotiated between the student and instructors. In many teacher preparation programs across the US, 
preservice student teachers develop these types of portfolios for reflection and to demonstrate their efforts, 
accomplishments, and progress in acquiring relevant knowledge and skills in a variety of teaching areas. In higher 
education portfolios are also used for accreditation reporting (assessment of learning). An eportfolio that is used for 
assessment q/Tearning generally links examples of student work directly to program standards. Assessment of 
learning portfolios are often part of an assessment management system at the university or college level, used for 
accreditation reporting or other assessment and research purposes. The content of these types of portfolios is 
generally not controlled by the learner, so they cannot be strictly thought of as student-centered learning devices in 
the same sense as a portfolio with negotiated or student controlled content. Portfolios used for assessment of 
learning are generally the data collection component of assessment management systems. 

The explosive growth of the Internet has ushered in the Information Age and added new terms to the 
educational discourse on learning. With the rise in the accessibility and use of networked computing, there has been 
an increase in the migration of learning and teaching tasks to computers (Siemens, 2005). This trend has fueled the 
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creating and disseminating of portfolios using computers, and increased the viability of the portfolio as a 
communication tool, with the capability of reaching a much wider audience than was previously possible. 
Electronic portfolios (a.k.a., eportfolios) have begun to replace the cumbersome paper/binder portfolios and are 
eliminating concerns over transportability and storage of massive amounts of data. An eportfolio is a purposeful 
collection of electronic documents and multimedia objects that is formatted as a website. Students can generate 
their work in various multimedia formats, such as video, audio, graphics, and text, and use hyperlinks for 
organization and navigation (Barrett, 2005; Curtis, Yanes, & McWright, 2003). Curtis, Yanes, and McWright 
suggest that eportfolios make the student work “accessible, reviewable, and replayable,” and address student 
ownership and storage issues. 



Student Needs in ePortfolio Use 

The shift to eportfolio development holds both blessing and curse. On the one hand, a digitized collection 
of artifacts enables a more efficient organization and review of past performance and facilitates the reflective 
process. On the other hand, as a stylized website, the eportfolio represents what Dillon and Zhu (1997) have termed 
the ‘newest genre,’ and necessitates the development of an entirely new set of skills related to web authoring. When 
an eportfolio functions as a student-centered learning device, as in the assessment for learning scenario described 
earlier, the benefits gained are contingent upon students’ meaningful use of the tool. Siemens (2005) notes that, 
although higher education can introduce the concept, a successful eportfolio initiative must be driven by the 
learners’ understanding of eportfolios as a tool for teaching and learning. Prospective teachers’ understanding of the 
usefulness of technologies as learning tools needs to extend beyond the development of skills for using specific 
software applications, and into the realm of pedagogical rationale for technology use (NETS-T, 2002). It is 
therefore implicit upon higher education to provide effective support for both the new software skills and the 
development of an understanding of the pedagogical rationale for its use. 

Considerations involved in supporting an eportfolio initiative in higher education are complex. Leon and 
Pearl Paulson (1994) describe the conflicting paradigms that coexist in eportfolio initiatives and the importance of 
separating the assessment management purpose (i.e., positivist portfolio) from the learner-centered purpose (i.e., 
constructivist portfolio) (Barrett, 2004). The positivist approach assesses student learning outcomes based on 
external standards, and assumes that meaning is “constant across users and contexts.” The constructivist approach, 
on the other hand, “puts a premium on the selection of items that reflect learning from the student's perspective” 
(Paulson & Paulson, 1994). Upon initial examination of an initiative, one or the other of these components may be 
less obvious, but both dimensions of eportfolio use must be supported at an appropriate level if the initiative is to 
succeed. 

Success with regard to the assessment management (positivist) purpose is evidenced by successfully 
accomplishing a goal dictated by external standards (i.e., accreditation of the institution; assigning a grade to student 
work). Elowever, success with regard to the student-centered (constructivist) purpose is more difficult to gauge. 

The level of success will depend upon how well the level of support matches the level of need. It is therefore 
necessary to identify the level of student need in any given implementation. In this regard, data on student 
perspectives are valuable not only in terms of evaluating, ex post facto, the success of the initiative in reaching 
learner-centered goals (i.e., summative assessment goals), but also to inform the design and ongoing improvement of 
appropriate support (i.e., formative assessment goals). 

Student Perspectives: Purpose, Support, and Personal Impact 

Support for the positivist purpose (i.e., to manage a comprehensive assessment o/ learning) at an 
institutional level is directly related to the skill of the assessment team and the professional level of support provided 
by the team members. Elowever, support for the constructivist purpose must address the needs of the student 
engaged in assessment for learning. Initially the student gauges the personal impact if the initiative fails to meet the 
constructivist goals. The level of support needed will depend upon what the student perceives as their stake in that 
purpose (personal impact) (Ritzhaupt, Singh, Seyferth, & Dedrick, in press). For example, if the student perceives 
that his or her stake in the purpose is high (i.e., a grade is dependent upon use of the eportfolio technology), then the 
need for support in developing the necessary software skills and understanding the rationale for its appropriate use 
will be high. If the student has very little to lose in the event that the initiative fails to meet its goal, then the needs 
do not exist and support is not necessary. For example, if the use of the eportfolio technology as a student-centered 
learning device is optional, the student risks nothing by not learning how to use it for that purpose. 
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Describing Success: ePortfolio Implementation Support Model 

The success of an eportfolio initiative can be described in terms of the tension between the diffusion of the 
technology (Strudler and Wetzel, 2005) and resistance to its use (O’Hara, Watson, &Kavan, 1999). Kurt Lewin’s 
Force Field Analysis (JPC, 2002) proposes that organizational forces are constantly seeking a state of equilibrium 

between support for change and resistance to change. Support 
for the adoption of innovation (i.e., diffusion) is in part based on 
personal impact to the stakeholders (Rogers, 2003). 

Figure 1 represents the structure of this relationship and 
conceptualizes this structure with descriptors for the four 
quadrants: High Diffusion/Low Resistance, High Diffusion/ 
High Resistance, Low Diffusion/High Resistance, and Low 
Diffusion /Low Resistance. Analysis of an eportfolio initiative 
should examine separately the assessment management 
(positivist) and student-centered (constructivist) purposes 
(Paulson & Paulson, 1994) and map the characteristic 
components (i.e. level of support/level of resistance) of the 
implementation onto the support model. In terms of the 
support model, a successful implementation will be 
characterized by High Diffusion and Low Resistance. This 
framework can be used to assess the various characteristics of an 
eportfolio initiative, and can be used to explain the outcomes of 
k)W Resistance to use h tf 1 those initiatives. 

The next section describes the study that generated the 
student perspectives data and the themes that emerged from the grounded theory analysis of an open ended student 
perspectives survey item (Ritzhaupt, Singh, Seyferth, & Dedrick, in press). That discussion is followed by 
application of the ePortfolio Implementation Support Model (ePISM) as a framework to explain the outcomes of 
two eportfolio initiatives. The initiatives described reflect two different approaches to eportfolio integration. Both 
the initiatives discussed were studied in the development of the EPSPI. 

Method 
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ePortfolio Initiative 

The eportfolio integration initiative described here was conducted in the College of Education (COE) at a 
major research university in the southeastern US. Two years prior to a reaccreditation visit from NCATE and the 
state Department of Education, the COE’s research and evaluation team identified a need to improve the 
reaccreditation reporting process. The necessity of retaining required performance -based assessment measures, 
while at the same time enabling the manipulation and aggregation of data, pointed to the use of an electronic 
portfolio and assessment system. 

A search began in early 2003-04 school year to identify an appropriate and affordable assessment 
management tool with an eportfolio component. The research and evaluation team narrowed the search to three web- 
based applications, and presented findings to a committee appointed by the COE Dean. The committee was made up 
of faculty representing each teacher preparation program in the COE. Of the applications examined, the Chalk & 
Wire ePortfolio © system provided the most powerful reporting package and appeared to have the most flexible and 
responsive support system for students. 

Since the system was to be used as an assessment management tool, it was imperative to incorporate 
national, state and local performance standards for each program. Research and evaluation team members worked 
very closely with faculty and department chairs to identify critical assignments in the curriculum that woidd 
demonstrate evidence of each standard, and to develop detailed rubrics with multiple performance indicators. 

A small cohort of Elementary Education students and faculty volunteered to beta test the system in spring 
semester of 2004-05. By the following fall semester 2005-06, there were six programs using Chalk & Wire 
ePortfolio ©. Faculty members were provided with an ePortfolio © account in which they could access the 
submitted student work and attach it to the corresponding rubric to assign grades and feedback electronically. 
Students were required to purchase a one year subscription for a cost of less than fifty dollars. Subsequent renewal 
subscriptions cost approximately half of the initial fee and were available after graduation. By the end of spring 
semester 2006, there were over 1,100 student subscribers, over 300 rubrics, and twelve sets of program standards. 
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A help desk support system was established in spring of 2004-05, staffed by three undergraduate students. 
Communication was limited to email, until the end of the spring semester when a small lab facility with six 
computers and a telephone became available. Help desk staff set up a series of orientation trainings, scheduled in the 
college’s open use labs, and facilitated periodic training sessions for both students and faculty. In the fall semester of 
2005-06, the help desk went online with a resource website, providing online training registration capabilities, and 
access to printable resource materials and tutorials. 

Procedures 

In the 2005-06 spring semester, a team of researchers adopted and tailored an instalment known as the 
Electronic Portfolio Student Perspective Instillment (EPSPI) (Ritzhaupt, Singh, Seyferth & Dedrick, in press) to 
evaluate the progress of the eportfolio initiative. The instalment was accessible in a web-based format and was 
accompanied with an open-ended survey item soliciting additional information “Please provide any additional 
comments, concerns, or suggestions”. The researchers posted a hyperlink to the instalment in the announcements 
section of Chalk & Wire ePortfolio © and sent an email with a hyperlink to the instalment to all students using the 
system. 

The survey was available for a three-week period, and during this time, two emails were sent to students. 
Respondents were informed that the purpose of the research was to: (1) monitor the progress of the eportfolio 
initiative, and (2) aid in the development and validation of an instalment designed to measure student attitudes and 
intended uses of eportfolios, namely EPSPI. Additionally, participants were informed that the survey was 
anonymous and that the information would not be divulged in any way. 

Participants 

Two hundred four of eleven hundred students completed the instalment anonymously. Of the respondents, 
approximately 95% were female. Approximately 50% of the respondents reported senior classification, 25% junior 
classification, and the remaining 25% reported graduate level status. Of the ethnicity of the respondents, 78% 
reported Caucasian, 11% Hispanic, 5% African American, and the remaining 2% reported either Asian or Other. 

The participants reported using the eportfolio system for an average of 9.77 months (SD=5.51). The eportfolio 
system had been in use for 18-months when the instalment was released. Ninety-one of the students responded to 
the open-ended item. 

Qualitative Analysis 

The open-ended survey item included with the quantitative survey provided faiitful information, 
complementing the quantitative survey. To systematically analyze the responses, three types of coding procedures 
were used in the grounded theory method (Strauss & Corbin, 1990): open coding in which the themes, or categories, 
and subcategories were identified and grouped, followed by axial coding in which the categories were restructured 
to show different relationships, and finally selective coding in which an emergent theory regarding the data is 
discussed. Open coding of the responses by two readers revealed four major themes: (a) system characteristics (b) 
support system (c) purpose (d) personal impact. Inter-rater reliability was calculated at .87. Identification of 
properties, or subcategories, provided detailed description of each theme. It was decided that the system 
characteristics theme (i.e., ease-of-use; support for application skills) was a subcategory of support. Thus, for 
purposes of this paper the four themes were collapsed into three: (1) purpose, (2) support and (3) personal impact. 
Based on the need to distinguish purpose (i.e., positivist or constiuctivist) in the analysis of an initiative, as 
discussed earlier, purpose was chosen as the central theme and an axial coding method was used to reorganize the 
other categories and restructure their relationships. Following is a discussion of the relationship between these three 
themes. 



Results and Discussion 

In the first theme, Purpose, participants expressed confusion and dismay at the purpose of the eportfolio 
system and process. Most participants clearly rejected the use of the eportfolio for seeking employment with 
statements like “I will never use it for future employers” and “I do not believe this is what potential employers want 
to see.” The subcategory ‘learning’ included mostly negative comments: “. . . not a good reflection of my work” or 
“many of the tasks that we are required to submit to Chalk & Wire are not examples of my best work”. 

In the second theme, Support, the majority of respondents felt that training and the help desk style support 
system did not meet their needs in various ways. These responses ranged from a general lack of awareness about the 
available support to inability of the system to anticipate and address diverse ways of learning. A major aspect of this 
theme involved faculty buy-in and support for student use of the system. More than 25% of the respondents 
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mentioned that faculty members did not offer help with using Chalk and Wire and did not appear to understand the 
program themselves. Another dimension of this theme indicated that the faculty did not value the system and many 
instructors required hard copies of assignments in addition to electronic postings. 

The third theme, Personal Impact, included responses about the investment in time and money, comments 
indicating predominantly negative attitudes toward use of technology, feelings of lack of choice and control, and 
other various emotional factors like “adds unneeded stress”. In summary, the qualitative analysis revealed an 
overall negative response to the eportfolio initiative. 



htfi 



9 

IS 



T 



ePortfolio Implementation 1 : An eportfolio initiative was piloted with twenty-six students enrolled in an 
introductory computer science course in a college of engineering at a mid-sized university in the southeastern US. 
Use of a free-form authoring system for developing an eportfolio was incorporated into the curriculum of this 
course, in which students created their eportfolios to reflect meeting the learning objectives in the coursework. Use 

of the eportfolio technology in this implementation was 
centered between the assessment management and 
constructivist purpose. For example, the eportfolio included 
a predefined structure containing elements that may be 
found on a resume, such as the student’s name and picture. 
Additionally, for each of the assignments, students were 
required to post a description, reflection, and a list of 
learning objectives. Thought the assignments had minimal 
expectations, the students had a high degree of flexibility in 
how/what they submitted to complete the assignments. 

Figure 2 represents the relationship between the 
degree of diffusion of the technology and the degree of 
resistance by the students. Because use of the technology 
was an integral part of the coursework, each student used 
the technology: High Diffusion. Every student used the 
technology as it was intended to be used (i.e., in creating an 
assessment portfolio): Low Resistance. The strength of the 
support structure can be discussed in terms of the two 

component parts: support for the development of (1) software application skills and (2) pedagogical rationale for 
technology use. In this implementation, there was strong support for the development of application skills, because 
the students developed the eportfolio in class with the assistance of the instructor. Likewise the instructor’s 
incorporation of eportfolio into the instructional activities provided strong support for pedagogical rationale for the 
use of eportfolio technology in assessment. Data from the EPSPI survey administered in this class indicate that 
overall the student reaction to the use of the eportfolio for the purpose of assessment management was positive; the 
implementation was successful (Ritzhaupt & Singh, 2006). 
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ePortfolio Implementation 2: The eportfolio initiative used for the qualitative analysis can also be 
reassessed using the ePISM as a framework. Use of the eportfolio system for assessment was established by the 
administration as a graduation requirement; faculty members were also required to use the tool in reporting grades: 
High Diffusion. Accreditation goals were met and the initiative appeared to be a success. However, results from the 
EPSPI indicated that approximately 20% of the participants were dissatisfied with the implementation of the 
eportfolio initiative (Ritzhaupt, Singh, Seyferth, & Dedrick, in press). In contrast to the first implementation, in 
which over 90% of the students said that they would use eportfolios to guide their skills development, 72% of the 
participants this initiative said that they would not use the eportfolio to guide their skills development (Ritzhaupt & 
Singh, 2006; Ritzhaupt, Singh, Seyferth, & Dedrick, in press). One possible explanation for this discrepancy comes 
from an examination of purpose. 

In this implementation, students were required to purchase a subscription to the eportfolio technology. 

Thus, the personal impact if the initiative did not meet the constructivist goal was relatively high (i.e. students 
perceived the purchase of the subscription as a waste of money). Student responses to the open ended item on the 
EPSPI indicate that the technology was not used as a student-centered learning device: Low Diffusion. Further, 
faculty resistance to the use of the technology other than for assessment reporting was very high: High Resistance; 
characteristically, the two support components (i.e., (1) software application skills and (2) pedagogical rationale for 
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f^»s 3 . hi00m0rtdion 2 conMaEtMri Puipoae technology use) were not in place. In terms of the 

constructivist purpose, the implementation appears to 
have been unsuccessful (See Figure 3). 

Closing Remarks 

This paper has analyzed student responses to an 
eportfolio initiative using grounded theory, and described 
three interrelated characteristics that emerged from a 
student perspective: (1) purpose, (2) support and (3) 
personal impact. Understanding of these characteristics 
and their interrelatedness is critical to the success of an 
eportfolio initiative. Additionally, this paper provides a 
parsimonious framework to assess the outcomes of 
eportfolio initiatives: the eportfolio Implementation 
Support Model (ePISM). The ePISM has two key 
dimensions (resistance and diffusion), which were used to 
explain two divergent instantiations of an eportfolio 
initiative. These results suggest that data on student perspectives are valuable not only in terms of evaluating the 
success of the initiative in reaching learner-centered goals, but also to facilitate the planning, implementation and 
ongoing improvement of eportfolio initiatives. 
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Troubleshooting Windows Movie Maker: 

Common Student Errors and Effective Instructional Solutions. 

Andrew J. M. Smith 

As part of the infusion of technology into schools, teachers frequently require their students to present a 
summary of their work in some electronic format. Often the media specialist is charged with teaching the various 
software products that allow students to present their work in new and interesting ways. Microsoft PowerPoint has 
been the favorite of many teachers, but increasingly Windows Movie Maker has been gaining popularity as a 
presentation medium because of its availability, its low cost, and the fact that it runs on the Windows platform which 
is generally available throughout the school and in students’ homes, rather than requiring a Mac platform. 

Such software products now give students and teachers enormous amounts of computing power, and the 
ability to perform tasks that until relatively recently required expensive editing suites and enormous amounts of 
expertise. While there is no substitute for expertise at the professional level, students are still able to accomplish 
many tasks satisfactorily, and to produce good quality products. 

However, students are prone to make certain errors in software manipulation. While some of these are 
specific to the Windows Movie Maker software package, many more of them are generic problems that surface for 
almost every software package, and which can determine whether or not the student is successful and has a positive 
experience with technology, or ends up frustrated and discouraged, vowing never to use the technology again. 

Bad technology experiences, particularly with video programs, can lead teachers and media specialists to 
question whether the extra effort in learning Windows Movie Maker is worth the bother. After all, video editing is 
too complicated, and we can accomplish everything with PowerPoint. In addition we don’t have enough video 
cameras for all our students to use, and, in any case, iMovie does it all better, doesn’t it? 

The truth is that the basic skills are not too complex for teachers and students to master and, while 
PowerPoint has become very powerful, there are still some things it cannot do, or perform as easily as the Movie 
Maker product. In addition, you do not actually require video footage to work with Movie Maker - still pictures can 
be used to produce very effective pieces, making use of the various transitions and effects to give the impression of 
movement. (The iMovie/Movie Maker debate will not be discussed here beyond saying there are some things more 
easily accomplished in one program, and others more easily in the other.) 

This paper is based on one teacher’s experience of teaching Windows Movie Maker over a three-year 
period at the college level to both in-service and pre-service media specialists. An examination of teaching notes 
from all three years has revealed a pattern of errors that frequently recur while students construct video projects, 
both large and small. 



File Naming and Saving 

Before any work can be done in the production of a new media product in Movie Maker, the students must 
have source material. This is in fact the source of the greatest number of problems encountered. This stems from 
three major problems: 1) a failure to understand file structure and how to save files, 2) the lack of understanding of 
the difference between a Movie Maker Project file and a finished Movie file, and 3) constant transfer of projects 
backwards and forwards among various computers, without knowing where the source files are saved. The most 
obvious manifestation of this error is when students open a project and discover a string of large red crosses across 
the screen instead of the carefully crafted movie they expected. 

The solutions to these problems are straightforward, and apply to many different software programs and 
projects. Firstly, it is most important that students learn how to save files properly. They need to understand the way 
files and folders are arranged, and how they can create their own folders. Too often teachers assume digital native 
students have a firm grasp of basic computing skills, but while they may be adept at manipulating various 
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technology and software tools, their basic understanding is frequently lacking, mostly because their skills have been 
picked up on the job through trial and error, rather than through a comprehensive, organized system of instruction. It 
is also worth remembering that most of these computer-based organizational structures are in fact based on old 
fashioned paper filing systems - familiar enough to the older generation, but unfamiliar territory to the digital native. 
Secondly, life is greatly simplified by teaching students to give files unique, descriptive names. It is much easier to 
help students work with files named, for example, “Autumn Leaves 2007” rather than “project 1”. Lastly, the single 
most effective antidote to the red crosses is to encourage students to create a new folder for each project, into which 
they put all captured video for the project, as well as all pictures and sound files they will use. Students should be 
strongly encouraged to assemble everything in that one folder before commencing the movie editing. In particular, 
students should be discouraged from importing or cutting and pasting from other sources directly into the program. 
The video creation process is made considerably easier by assembling all materials together, then importing them all 
into the same Movie Maker collection, to simplify creation of the final product. (A particular danger spot is when 
students are capturing video from a source directly into the computer, as they are often unaware of the default target 
folder to which the video will be saved.) 



Selecting the Source for Insertion 

Although students may create a folder to assemble all the component parts, they do not always take care 
when importing those materials into a collection, and frequently import video from the default capture folder, rather 
than the project folder they created. This will work perfectly well if the entire project is completed on that computer 
and the movie finished at that time. However, if the work is saved as a project and then moved to another computer, 
the target files will not be identified and the red crosses will appear in the project. Again this highlights the necessity 
for students to pay particular attention both to where they are saving component files - videos, pictures, and music - 
in the first place, as well as to the source from which they import the components before adding them to the actual 
project. (Movie Maker does have a feature to allow you to search for missing files, but it is preferable for the 
students to understand the process properly from the beginning, so this is not required.) 

It is also imperative students understand the difference between a Movie Maker project and a completed 
Movie Maker video. The project is simply a set of instructions referring to the various videos, pictures, sounds and 
other elements. It previews how the finished video will look and sound, but everything is changeable until the movie 
is finalized. When various elements are inserted into the storyboard or timeline, the program only inserts a 
placeholder there. The actual material still resides in the source folder, and is previewed as needed while working on 
the project. Only when the video is finished or finalized are the various components actually brought together or 
rendered into the actual video file. It is this rendering process that can take some time, and explains why the process 
of saving a project is almost instantaneous, but the process of finishing a movie takes significantly longer. The 
project file therefore contains only placeholder and editing information (e.g. insert clip 1 here, fade in, use overlay 
title “Autumn Flowers” etc.) while the video file contains the actual video footage, audio, pictures, titles and 
captions in finalized form for showing on a computer or other display device. 

Planning and Process 

Many more difficulties arise if students are not encouraged to plan their projects in sufficient detail before 
commencing the software manipulation. Common problems caused by lack of planning fall into two major 
categories: lack of materials or inappropriate materials (copyright and fair-use wise) and editing problems based on 
sequence of editing. Students frequently embark on projects by focusing on the technology itself, rather than the 
desired end product. This may be a function of how we teach new technology products, although there is always the 
novelty factor of the new software as a distraction, even if planning is emphasized. If the focus is on the technology, 
then little thought is given to planning, including scope, duration, required resources, and expectations. However it 
is these elements that are crucial to the success of any video project. It is helpful to remind students that while the 
software affords them unprecedented technical opportunities, it does not automatically grant them film-making 
expertise. This is a learned skill, and just as any motion picture starts with months of careful planning long before 
any film is shot, so the simplest Movie Maker project requires to be thought through insufficient detail to ensure the 
project is manageable within the available time, all component materials are identified and located, and the process 
can be finished in a professional and timely manner. Storyboarding is a useful aid to setting up any video project. 
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but even simple projects using only still pictures will run more smoothly if an outline is created and materials 
identified before any actual software editing takes place. 

Identifying and obtaining materials can also be areas of complication. Original pictures, video and music 
can present problems of access or composition for students although they are generally free from copyright 
problems. Often component materials can only be obtained from other sources, and in these instances care should be 
taken to ensure materials used are royalty-free or may be used under the fair-use guidelines. Particular attention 
should be given to the end purpose of the students’ videos. If these will be posted on the web or in some other public 
access forum, then copyright issues must be addressed. The ease with which copyright materials can be obtained, 
copied and reused does not release us from the obligation of teaching our students the ethical use of such materials. 

A checklist of all required materials can be a useful aid to ensure all necessary components are assembled before the 
software portion of a project is begun. Checklists can detail general or specific requirements for videos, pictures and 
sound, and can also serve as shooting scripts to help students identify which shots they will require to complete their 
projects. A useful exercise is to give students a specific topic to research, then to give ten minutes for research and 
planning, ten minutes for component gathering, and ten minutes for software manipulation. This encourages students 
to think ahead about what they will need, to identify available resources, but most of all, to work with the materials 
they have. This above all else demonstrates the necessity of research, planning and organization. If there is no plan 
and materials are not there, there is little substance to the project, but even a small number of materials in a well- 
executed plan can result in a quality project, albeit limited in scope. 

The sequence of editing events within a project can also cause problems for students. This most frequently 
occurs when students are working with separate audio tracks and complete all video and sound matching editing 
before thinking about titles and captions. Often the carefully synchronized materials are displaced with captions and 
titles and much hard work is lost. Again it is worth reminding students of the order in which professional filmmakers 
work, where sound is added after the visual portions are edited. In this case students need reminding that the titles 
and captions are visual elements as much as pictures and video clips, and require to be inserted before final music or 
soundtrack adjustments are made. Another technique that may be helpful is composing a project in smaller pieces. 
Movie Maker is a fairly straightforward and basic video editing tool, but although it lacks many features of 
professional editing suites, we can accomplish more complex projects simply by changing how we use the software. 
Instead of trying to complete many complex synchronizations of soundtrack or narration with many video clips or 
pictures in one project, smaller component parts may be finished as movies in their own right, then brought back 
into the project collection and inserted into a new project as whole pieces. In this instance the addition of new titles 
or the repositioning of various elements does not affect the formatting of the intact component pieces. This is also an 
easy way to add an additional audio track, if required. The original movie with video sound and separate audio track 
can be finalized as a video, thus combining all the elements into the one track. If this is now inserted into a new 
project all previous elements are contained within the picture and audio sections of the video track, and a new, blank 
audio track is now available (as are new title tracks and so on.) 

One last word on planning must be said. All video projects take more time than we think to complete well. 
Even if the students are technologically savvy, there must be sufficient time allowed for research, planning, 
materials collection and project assembly. Care should be taken to ensure that teachers’ expectations are realistic, 
given the available time for these activities. In addition, thought should be given to whether students can gather the 
required materials easily, safely, quickly, and legally. Projects with a smaller scope and well-defined expectations 
generally provide a better learning experience. 

Shooting Video and Audio 

There are a variety of common mistakes students make while actually shooting video. While there are a 
variety of excellent books which give detailed instructions on how shoot good video, there are some basic flaws 
instructors should highlight to minimize the distress to students when they discover they have unusable footage. 
Before any shooting takes place a plan must be created that details all the various elements the students will need for 
their final projects. Without a plan or a checklist, the process is simply haphazard and essential elements will be 
missing, while extraneous footage will be shot. Perhaps the most common error here is not giving enough lead time 
for the camera to start filming before the interviewee talks or the event happens. Most cameras do not start filming 
immediately the start button is pressed, but take a few seconds to warm up and then fade in to the picture. If there is 
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no lead time on the raw footage, it is extremely difficult to make satisfactory edits on the finished product, and the 
editing becomes jumpy and disjointed. Although we don’t want to waste digital video tape, a good ten seconds at the 
beginning and end of every shot is a wise investment for the editing process, and saves a poor result or the necessity 
of shooting the video again, which is not always practicable. 

Light, or lack thereof, is the next most common shooting error. Students must be taught to understand that 
while the human eye can distinguish detail in varying patterns and intensities of light, video cameras are much more 
limited in their ability. More light is usually required indoors to ensure the picture quality is adequate, and it is often 
helpful to encourage shooting to take place outdoors where there is an abundance of natural light. (This, after all, is 
why the very early movie studios set up shop in southern California.) However even outside, care must be taken that 
light is shining directly on to the subject of the video from the front and that the light is not behind subject, causing 
it to be in silhouette. A worthwhile exercise is to have students fdm short sequences in different parts of a building 
and in different locations outside, so they can experience the effects of different lighting on the quality and clarity of 
their videos. Again student scan be reminded of the correct sequence of events by thinking of the classic 
professional movie line, “Lights! Camera! Action!” which illustrates what is required to shoot video correctly. 

Another area of difficulty for novices is that their video footage is either shaky or out of focus. If there is an 
auto focus feature on the camera, make sure it is turned on, as this will enhance picture clarity. However if the 
camera is moved about too quickly, the auto focus will not be able to help, for the auto focus needs a few seconds to 
register what is in the viewfinder and bring that into focus. If the camera is sweeping or panning too rapidly, the 
camera loses track of what is in the viewfinder and focus is lost. This also means that the camera should be held 
steady for several seconds focusing on one thing, before moving. As well as helping with the focus problem, this 
also allows the viewers to develop a context for what they are seeing, rather than being disoriented by rapid camera 
movement. In addition to the auto focus, most cameras also include an anti-shake feature, which can lessen the 
effect of movement by the camera operator. However it cannot completely compensate for shaky hands, so student 
should be advised to use tripods or monopods, or to practice other stabilization techniques if neither of these is 
available. Indoors cameras may be placed on tables, shelves or any level surface, or the camera operator can brace 
him or herself against a wall or a piece of furniture. Outdoors trees, street furniture, and walls can all provide 
support, and if all else fails the camera operator can brace the elbows against his or her own body to reduce camera 
movement. Each of these techniques can be effective in reducing shaky or out of focus video, and they are easy to 
implement as long as students are made aware. 

A last but equally important area of difficulty in shooting video occurs with sound. In most cameras the 
built-in microphone is simply not adequate to capture sound from more than two or three feet away, and the 
resulting video footage contains much more ambient noise than the desired audio. In such cases an external 
microphone can make a substantial difference to the sound quality. Even a very basic microphone can make the 
difference, which is based more on proximity to the desired sound source than on the quality of the microphone 
itself. Note however there appears to be a trend in new, lower priced video cameras to eliminate the external 
microphone jack, thus removing this easy solution. Media specialists should take care when purchasing video 
equipment that this necessary feature is included. 

Software Manipulation and File Size 

Once all suitable materials have been gathered, there are traps for the unwary in the editing process. 
Students often fail to recognize the difference between the storyboard view and the timeline view and the different 
tasks they can accomplish in each view. While everything can be edited within the timeline, the storyboard gives an 
easier interface for adding transitions and effects, particularly for younger students. The storyboard also gives an 
excellent pictorial overview of the project and may be used to identify missing transitions or effects. The timeline 
can be more complex, and students often encounter difficulty by endeavoring to work on the timeline at too low a 
level of magnification. For there are helpful zoom in and zoom out buttons which allow the timeline to be expanded 
for very precise editing or to see fine detail or reduced for more global edits or to see the bigger picture. Both the 
timeline and storyboard views will be required in the course of even a modest project, and students should be 
encouraged to experiment with them, discovering for themselves which editing tasks are more easily performed on 
each. 



309 




A final problem area concerns saving the final project as a movie. This ties back to the idea of students 
knowing the difference between a Movie Maker project and a finished video. If students are working with all their 
component parts inside a project folder on their own USB drive, they must understand that the process of finalizing 
the movie will result in the creation of a new file at least as large as all the component pieces they have inserted into 
it. It is at this stage that things go wrong. Often students are working on USB drives that are full to overflowing and 
have not the capacity to save a file the size of the finished video. It is especially crucial at this stage that students do 
not delete the component parts of their video (since they have already used them in the video and don’t need them 
any longer!) before the movie has been finalized or rendered. A good solution is to have students finish the video to 
the hard drive of the computer on which they are working, and then, only when the completed work has been 
checked, space is cleared on the USB drive and the completed work saved. Note that this method destroys the source 
material and will preclude easy edits to the project, should they become necessary. Many computers offer the option 
of finalizing movies directly to CDROM or DVD ROM, but, in practice, this seldom seems to work, and it is safer to 
finish the movie to the hard drive and then copy it to a new storage medium. 

Finished size and quality can also problem encountered at the end of the project. Students must be taught 
the relation between file size and video quality and that videos designed for tiny personal players with two inch 
screens will not look nearly as good on large computer monitors. In addition, it is often at this stage that students 
realize that some of the materials they selected for inclusion started out with lower quality, which is only 
exacerbated by the compression techniques employed in rendering video. In particular, still or video camera settings 
may be changed to allow for the capture of many more pictures or minutes of footage, but there is always a cost, and 
that cost is quality. If quality is compromised early in the process, there is nothing that can be done at the editing 
stage. However higher quality (and therefore larger) components can always be reduced or compressed more during 
the final editing stages, with less sacrifice of clarity. 



Summary 

While it may appear video projects contain too many hazards for students to negotiate, software such as Windows 
Movie Maker offer an excellent tool to students of all ages for assembling and presenting information effectively. 
This paper has endeavored to highlight some of the most frequent student errors encountered with this software, and, 
by offering some solutions to these problems, allows media specialists and teachers to prepare students ahead of 
time, so as to minimize their impact. At a time when technology literacy is increasingly assumed to be present, there 
are large holes in students’ basic understanding of how software operates, and the most effective way for them to 
interact with it. The more we can identify the problems and offer effective solutions, the better prepared our students 
will be, and the better they themselves will be able to use the technology tools available to them. 
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SDAIE Online and Intercultural Competence: 

Theory into a More Flexible Practice for Course Designers 

Mari Vawn Tinney 
Western Governors University 

Abstract 

This research proof of concept design study examined how this researcher adapted the SDAIE 
(Specially Designed Academic Instruction in English) model as a guide for designing cross-cultural online 
learning experiences. SDAIE is designed to overcome cultural and language learning barriers in academic 
courses for international/diverse/ESL students. This intentionally designed holistic case study explored the 
benefits of applying a conceptual design tool, SDAIE, to an online environment. The primary research 
question for this study was: In what ways can the same benefits of a low-risk and safe environment in a face- 
to-face (F2F) SDAIE course design be realized in an online course design? An actual SDAIE Online course 
was designed, developed, and instantiated with a specific focus of using the SDAIE components of low-risk 
and a safe environment for seven Hispanic and Latino college age bilingual learners. Data was collected on 
each of the participants during a five-week period. The findings are reported from participant responses 
given during a formative evaluation. Data from the study are presented in tables, participant narratives, and 
investigator comments. Participants’ favorable response to the SDAIE methods and strategies suggest that 
these learners received similar or some of the same benefits while participating in this SDAIE Online design 
instance as they would in a SDAIE face-to-face course as it was implemented with three SDAIE 
subcomponents. Design recommendations are given for the intended audience of this design study, who are 
instructional designers, course developers, and instructors, that they may discover the stage of intercultural 
competence they are use in becoming more culturally sensitive in their views, lives, and work applications. 

Introduction 

With improved software and global connections, instructional designers encounter more 
international students and diverse learners as they design and export course content via the Internet and 
blended courses. Due to the need of being aware of the difficulties of designing instruction for students from 
various cultures, the impact of culture in instructional design practice should influence how design is 
approached. Many designers are currently unable to develop instruction that accommodates different 
cultures well. Cultural sensitivity is not exhibited in the technological tools as much as it is in the 
pedagogical process of using a variety of models to design and implement instruction (Chen et al., 1999). 
The area of training instructional designers to gain cross-cultural competence is still in its infancy in 
designing culturally responsive interfaces, lessons, and course elements. Neither educational courses offered 
by designers nor thinking skills of students are culturally neutral (Kern, 2006; Singh and Pereira, 2005). 
Many are unable to develop instruction that accommodates differences in the various worldviews, 
expectations, living and learning styles, or preferences of diverse academic or work group expectations. 
Therefore, research is necessary to find designs or models that allow for effective learning across cultures 
(Henderson, 2007). 



Literature Review 

In looking for a beneficial design tool for learning across cultures, I considered what designers 
could use to adjust content and create online materials for learners who represent various cultures, contexts, 
and viewpoints that vary from the mainstream. To solve some design problems that influence online course 
culture shock, dropouts, and cultural insensitivity in courses, my research and development project focused 
on use of a design study (Design-Based Research Collective, 2003; Gibbons & Bunderson, 2005). Three 
basic approaches for designers to use in knowledge-producing research are to explore, to explain, and to 
design. This design study involved adapting a face-to-face (F2F) tool and model to eLearning use. I selected 
the F2F conceptual tool, SDAIE, which is pronounced Suh-DYE. SDAIE is a conceptual design tool and 
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pedagogical tool, “an evolving instructional process that includes teaching content and English language 
development simultaneously” (Cline & Necochea, 2003). 

SDAIE was implemented 25 years ago to overcome cultural and language learning barriers in 
academic courses for bilingual, international, and English as a second language students (Balderrama & 
Diaz-Rico, 2006; Krashen 1981). “California developed the term SDAIE to draw a distinction between 
programs that provide rigorous academic curriculum and those which do not” (Sobul, 1995, p. 7). For 
example, teachers in California are required by law to apply SDAIE methods in their courses, pass a SDAIE 
certification test or hold bilingual education certification in order to teach. By combining factors such as 
international learners’ and ESL students’ needs with the power of applying F2F SDAIE methods to an 
online environment, I coined “SDAIE Online”. 



Method 

This SDAIE Online research involved creating the design, development, and implementation of a 
prototype computer-assisted language learning (CALL) course as a proof of concept. The intent of this 
exploratory design study research with qualitative methods was to offer designers more design options based 
on SDAIE methods and strategies. The applied research approach studied an online phenomenon that did not 
exist yet but was instantiated a small pilot study as a design case and was used with bilingual Latino 
speakers of Spanish and Portuguese. This effort fell into the category of an intentionally designed holistic 
single case study or a “design instance” (Reigeluth & Frick, 1999). Another purpose of this design instance 
was to help international students interact with content that often requires more knowledge of the cultural 
and educational schemas that designers assume learners already know (Stewart & Bennett, 1972; Valdes, 
1986). 

SDAIE has seven main components (Cline & Necochea, 2003), but the research focus was on one 
SDAIE component. Since most of the other SDAIE components have been researched and tested with 
results (Chamot & O’Malley, 1994; Echevarria & Short, 2004; Echevarria, Vogt & Short, 2000; Hinkel, 
2005), I focused on the component of low risk and safe environment. I chose this for my course design and 
data gathering because there is a gap in the research with no direct results from a proof of concept 
application on the benefits of such an approach as in my research study. The research methods included 
gathering various types of data (interviews, observations, assessments), which were analyzed for the holistic 
case study. The formative evaluation utilized observations, homework response documentation, 
questionnaire results, and inteiview statements taken before, during, and after the implementation phase of 
this study. 

The primary research question for this study asked in what ways can the same benefits (Hinkel, 
2005; Kern, 2006; Warschauer, 1999) of a low risk and safe environment in a F2F SDAIE course design be 
realized in an online course design? This research question guided a related research question: What does 
implementation look like in an online ESL learning environment when designers add three SDAIE design 
sub-components that create a low risk and safe environment in a F2F setting? These sub-components 
include: (a) guides and learning companions, (b) scaffolding for a low affective filter, and (c) short initial 
oveiview and technical training lessons on using online lesson components and navigating the system. It is 
usually more problematic for ESL students who take F2F and online courses to complete courses without 
these components (Aykin, 2005; Gunawardena, 2003; Palloff & Pratt, 2003). 

To examine this research question of designing a low risk, safe environment, I implemented the 
SDAIE Online course, “Communicating in North America”. The course was uploaded to a professional 
hosting website in Texas. As the subject matter expert and course designer, I was also the instructor for this 
SDAIE Online course, and the investigator of this study. I closely monitored and collected data on each of 
the participants during a five -week period. The context and content of this SDAIE online course are about 
learning the aspects of intercultural communication while adapting to a second culture (i.e., life in the USA 
or Canada). The lesson content deals with problems the ESL students have in common: learning reasons for 
classroom culture shock and making changes they may expect when taking a Western-style course or 
working on a job. SDAIE methods and strategies help students learn cognitive-academic language 
proficiency and subject area content skills (such as math, social studies, science) in context and with native 
speakers in the class or on the job. The course objectives and content for this design study are based on the 
research questions and the three SDAIE design sub-components that create a low -risk, safe environment. 
Designers must remember one important aspect in regard to applying SDAIE methods and strategies: the 
curriculum and objectives remain the same as in designing for traditional courses. Only the methods or 
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strategy presentation changes for the content in a SDAIE course. This difference is true of both the SDAIE 
F2F and online courses to make the content comprehensible. 

Table 1 

Components of the SDAIE Conceptual Model or Tool 



SDAIE F2F 



How SDAIE can be applied online 



Connect to previous learning. Especially begin 
lessons or new content chunks with activity or 
strategy to connect students to what they know. 
Build schemas related to the lesson content. See 
Big Picture. 

Use lots of realia, visuals, technology, and 
manipulatives that help ELLs in acquiring English 
and also connect words to concrete life and 
processes. Hands-on experiences. Multiple 
demonstrations in different contexts. Key word 
support list. 

Prepare a low-risk and safe environment. Less 
stressful. Use comprehensible input. 

Use Lesson design, delivery, and learning 
strategies that involve learners in authentic content 
tasks such as methods to develop higher-level 
thinking and language skills. 

Have multiple access points. Many strategies 
(lesson design, delivery, and learning strategies) to 
involve learners in authentic content tasks like 
methods used to develop higher-level thinking and 
language skills. Guided practice. Concept 
mapping. 

Make learning cooperative and interactive. Small 
group work. Mentors, aides, or teachers guide. 
Learning companions help. Can be a problem- 
based instruction model. Task- focused lessons 
incorporate specific language objectives 
(L/S/R/W). 

Include chunking and webbing. Use a content- 
based instruction model with content modification 
so that ELLs know what is expected of them. 
Integrate smaller chunks of information. Use of 
graphic organizers. 

Be respectful of learner as a person. Include 
family, cultural background into learning 
environment. Be patient for responses. 



Use of checklists of what they already know. Let 
them opt out of modules because of previously 
passed off competencies. Students can click 
through lessons for the overview and return to 
parts to study. 

Use electronic interviewing. Links to URLs for 
specific content info, visuals, videos, authentic 
language contexts. Independent and 
interdependent skills grow as technology skills 
increase in researches, games for language and 
strategy use. 

Let students take more time to compose answers 
online and use an alias (safer). 

Give students endless chances to practice with 
fast feedback. Computer lesson 
software/modules can model concepts. 

Computer simulations help. 

Add online support or scaffolding with 
hyperlinked text in readings can. Computer 
lesson software/modules can model concepts. 
Computer simulations give extra edge. 



Include all students in highly interactive 
discussions. Cooperative group activities in 
asynchronous conferencing, round robins, 
symposia or expert panels. Web buddy, jigsaw, 
group problem solving, email pals. Much 
reading and writing. 

Include pro-con discussions and presentations of 
content, articles, cases, and issues. Critical 
thinking activities happen in chats, threaded 
discussions with reflections, reactions, 
summarizing, tips. 

Use visuals, virtual characters, audio voices, 
acceptable colors and screen graphics, or content 
from cultural norms of learners. 



The content for the created blended lessons focused on the following: preferred learning styles, 
communication styles, cultural bumps people encounter, and becoming aware of the influences of high-and 
low-context communication, the influence of language families on thinking patterns and how these 
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difference matter when writing an academic paper or work report in the U.S., learning strategies, culture 
shock, cultural bumps (Archer, 1986), six stages of viewing cultural differences (Bennett, 2007), perspective 
transformation and attribution training (Furnham, 1993), and transformational learning (Mezirow & 
Associates, 1990). The design overview plans for the SDAIE online course are found in Table 2. I created 
this plan from combining my previous SDAIE course design knowledge, from lesson plans I had used to 
teach previous university courses, and from Merrill (2002) and Bonk & Dennen (2003, p. 344). 

Table 2 

Design Methods/Strategies for Use in this SDAIE Online Course 



Design methods and strategies 
Guides 

Use of virtual guides and 
learning companions who 
model language and 
behavior in social and 
interactive situations. Will 
be: human study buddy or 
digital voices, photos, and 
3D agent heads and bodies. 



How method is used in course 



Learners see and hear examples 
of virtual guides as models so 
learners may use language, think, 
act as natives do in real world for 
comprehensible input. Roles of 
guides: instruct, demonstrate, 
motivate, and introduce, conclude 
lessons. Humans interact with 
learners in assignments. 



Purpose of doing this 



As examples, guides enable 
meaning and skills for 
learning through connections 
in context for content or 
language. Guides strengthen 
online socialization and 
different ways to communicate 
through the Internet. 
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Scaffolding 

Use scaffolding for a low 
affective filter (less stressful 
situations) so learners do not 
feel as alone and can 
practice skills in 
environment safer from 
social ridicule. 

Safe Environment 

Create safe environment for 
interacting with access 
methods and technology 
required to navigate the 
course and do lessons. 
Familiarization with course 
and course policies. 

Cultural Insights 

Offer insights of the 
invisible cross-cultural 
realities and differences in 
preferences because of high 
and low context cultural 
views and communication 
styles. Thus, get perceptions 
of less stressful lessons. 



Various help methods are built 
into lessons, such as: tutorials, 
FAQs, virtual guides, glossary, 
instructor orientation to learning 
style awareness, help to learn and 
practice new learning strategies, 
predictable course structure. 



Complex tasks are broken into 
small safe stages to do whenever 
students desire. All students are 
required to do Pinnacle’s five 
intro tutorials: how to register for 
a course, use a discussion board 
and chat room, navigate the 
system, and take assessments. 



Eye opening video clips and 
examples are used so learners can 
examine their own stereotypes 
and get more intercultural 
knowledge. A few training 
exercises in intercultural 
communication and sensitivity 
can result in empathy for others 
and adapting to diverse situations. 



Having support from people, 
technology, and guides is less 
stressful when students stretch 
with assistance as they move 
beyond the limits of their 
abilities to a new ZPD. 



For increased confidence in 
using the course system and 
less stress over technical 
worries, learners practice how 
to interact with and use the 
main course system elements. 



Once students realize 
their high or low context 
views and expectations are 
causing frustration in 
situations at work or classes, 
they can find ways to be 
flexible and adjust to 
bicultural views. 



The instantiation in the course design for the first SDAIE sub-component, guides, focused on the 
use of virtual and human guides in a supportive affective environment. I believe that some cultural 
preferences and views can be altered by mentors, teachers, online guides, learning companions, and 
facilitators who act as transition guides. Implementation examples of what the SDAIE component of a low 
risk environment looks like with guides from the online course offered places in the course design for guides 
to act in instructional roles. The purpose of these guides is to assist the learners to make cultural transitions 
and gain confidence in their abilities to express themselves. These guides included human study buddies, 
low-tech voices, and digital 3D characters. Guides or near peer buddies and personas help students break out 
of the old image they have of themselves as learners. No matter what age or level of professional expertise 
learners attain, they can all benefit from guides and role models. 

The instantiation for the second SDAIE sub-component, scaffolding for a low affective filter in 
second language acquisition, came in the form of discussion boards, checklists, design details for Latino 
preferences, and online stars added for completed work, along with helps and tutorials. 

The instantiation of the third SDAIE sub-component offered a less stressful course through initial 
technical training tutorials. These tutorials are on how to register for a course, use the discussion board and 
chat room, navigate the system, take assessments, and check a user’s readiness to do online courses. 

One purpose of designing opportunities where learners could use these three subcomponents in the 
course was to allow experiences where people can find a new comfort zone by taking on a new persona until 
they could feel more comfortable in situations and reach their personal goals. Some learning companions in 
the course were the other students in the course who join in collaborative activities to achieve desired 
outcomes. 
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Some SDAIE Online Course Themes and Objectives 

One course objective was to enable learners to apply transformational steps to become increasingly 
bicultural and interculturally competent as they move between cultural and linguistic worlds. While working 
with others and the course guides to build new intercultural communication skills and views, these new 
experiences and perspectives enabled further transformation more easily and quickly. Because people are at 
different stages on the continuum of intercultural competence, they will see and react to the same situation in 
different ways. Examples and exercises in the course guided the learning process by which the participants 
were able to have their perspectives transformed. Each participant chose a cultural bump or surprising 
cultural situation in his or her life to examine, explore, and deal with during the month. Their reflections and 
actions became a part of “cultural reflectivity”, which is the process of engaging in critical examination of 
one’s own assumptions and of valid alternative ways of knowing and being. This transformation process 
allows one to become more aware of one’s own culture through a process of a more open-minded dialogue 
and interaction with different cultures. Cultural introspection and awareness helps students and professionals 
to become increasingly sensitive to differences that exist and increasingly skilful at understanding multiple 
perspectives behind cultural differences they notice. Another objective and course theme was that of 
encouraging mutual understanding and the development of new intercultural communication skills. In this 
design research, I outlined how Bennett’s (1993) six stages of developing intercultural competence (as shone 
in Table 3) would apply in the participants’ lives. Each learner was to notice their reactions and thinking 
throughout the last week of the course to ascertain which level of intercultural competency they were 
experiencing. 



Recognizing Levels of Cultural Competency in Life and in Instructional Design 

The same content that the Hispanic/Latino participants interacted with in this SDAIE Online 
course, in applying process of intercultural transformation in their lives, could also be useful to instructional 
designers and instructors. Professionals often do not recognize what stage they are at themselves in regard to 
the conditions and views of cultural differences. Information in Table 3 may help everyone understand 
themselves and their reactions to cultural differences better. These insights can also aid in understanding the 
reactions of others who might be in different stages of development. In Table 3 I have adapted information 
from Pang (2005, pp. 138-141), from content in a lecture I attended given by C. Compton (personal 
communication, October 29, 2004), and my own experience in working with designers. The table may be a 
guide for designers who might wonder which of the six stages they could be at in the continuum of 
becoming interculturally sensitive designers. I offer a description of what the six stages could be like for 
them in dealing with cultural differences during the design process as I have adapted them from Bennett’s 
stages. Table 3 can also be used by people who do not often realize that others are in different stages of 
development in intercultural adjustment, their level of cultural sensitivity, ethnocentrism, stereotyping 
others, racism, bigotry, or in becoming bicultural. 

Table 3 

Adapting Bennett ’s Model for Becoming Culturally Responsive Designers 



Description of stages 



Stages of becoming more 
culturally responsive designers 



Representative comments and 
responses during stages 



1. Denial: ignore some realities, 
feel ethnocentric, and have a 
belief that how life is in their 
cultural view is the only true 
view of how things should 
be. There is little exposure to 
cultural differences, and 
people mostly just glance at 
the surface. 



After going to another location and culture, 
people who are raised in a mono-cultural 
environment do not realize that they are still 
seeing things with the same worldview as 
they had back home. They think everyone 
sees with their worldview. This can also 
happen while working virtually with an 
audience from a distance when designers 
assume the learners can use the technology 
and the lessons the same way anywhere in 
the world. 



"What difference? This is just like 
what we have been doing all along. I 
don’t see any difference.” "They 
speak English, don’t they? Our 
content is fine as it is, and we’ll save 
money from doing translations.” “If 
they got hired into our company, 
they must have passed the 
requirements to understand this and 
can do it just like anybody else.” 
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2. Defensiveness: feel 

denigration from threats of 
cultural differences. Do 
broad negative stereotyping 
that is based on social 
categories such as race, 
cultural group, or age. 



Seeing the world as “us” and “them.” 
Frustration sets in when practices in some 
cultures may look the same on the surface, 
but the learners respond differently than 
expected to instructions. The cultural 
differences people notice make them feel 
uncomfortable or even sick from culture 
shock. 



“They are uncivilized and ignorant.” 
"He so polite that I do not know 
what his opinion is.” "They do not 
respond to my email.” “They are 
messy.” “My design is just fine and 
I'm not changing it to cater to them.” 
"Our way is superior. Their way is 
inferior.” 



3. Minimization: treat cultural 
differences as if they do not 
matter. The differences are 
minimized and do not seem 
to matter much. This is a 
common stage where people 
generally get stuck and do 
not move on to becoming 
culturally sensitive. 



After moving to a point of recognition where 
they know they should not he defensive 
about cultural differences people are aware 
of them and are defensive anyway at some 
level. So instead, people try to cover up the 
differences with looking for aspects that are 
similar. They focus on what is similar 
because they think or hope that we all 
experience the world in universal ways. 



“We are all in this same profession. 
We have so much in common.” “If I 
wait long enough, they will begin to 
understand what we are doing here 
and see it the same way I do.” "We 
are all basically alike as humans.” 
“We are all children of God.” “We 
all learned to design with 
constructivism, so it’s fine.” 



4. Acceptance: have an attitude 
of accepting differences in 
more contexts. People begin 
to see a continuum of 
complex worldviews. They 
do not see issues as black 
and white. They use more 
non-verbal cues. 



When encountering cultural differences, 
people are not offended. Rather, they simply 
accept the differences with respect, but not 
more than or less than their own ways. They 
see others as different from themselves and 
the world as a more complex place to 
navigate. 



“Well, that’s different,” designers 
finally realize. “Oh, how interesting! 
I wonder how this happens?" "I 
accept these differences.” “I wonder 
what this means to these people?” “I 
celebrate a chance to use brighter 
colors in these slides.” 



5. Adaptation: develop new 
skills sets and empathy to 
use these skills according to 
context. This stage is more 
living, doing, and application 
by adjusting to different 
cultural ways than only 
thinking about cultural 
differences. 



Adapting is not assimilating. Instead it is 
learning to see, feel, and believe as the other 
does in different contexts as they come 
along. Bennett wrote that in this stage 
culture is understood as a process we engage 
in. It often takes more than five years of 
living in another culture to really begin to 
adapt and shift thoughts and actions in 
authentic ways. 



“This is just different, and I can do 
this process either way.” “Let’s shift 
our framework so that the learners 
can view this problem from several 
contexts.” “It will be worth it to 
adapt the interface so it is localized 
for this large segment of our 
audience.” “This graphic is so 
perfect for this, and they will relate!” 



6. Integration: be exposed to so 
much change people are not 
sure of their own identity 
with any one culture. 



When people start blending in as part of one 
or more cultures, they get a sense of 
disorientation and who they are. They use a 
variety of worldviews and customs. 



“Don't ask me to edit this piece. My 
English is not so good anymore.” 
“Will they even like that design if we 
use it?” 



Bennett’s Stage 4 and stage 5 are much more preferred while persons adjust their lives and 
worldviews as they work from the acceptance stage between two or more cultures. In these stages, people 
can be seen as functioning in multicultural, intercultural, and multi-contextual frameworks as they build the 
bridge and are well connected in at least two directions. A common response is one of relief when learners 
realize that it is not necessary to remain in any given stage Learners can leave their comfort zones by 
reflecting on their sense of reality in the continuum of any of the six basic stages. 

Findings 

From the three SDAIE subcomponents that I designed into the course, the research findings 
revealed seven areas where applying SDAIE methods and strategies benefited the learners. I noted 
participant’s responses week after week during our formative evaluation interviews regarding the basic 
course elements, objectives, activities, concepts, and the way they were implemented in the course. It was 
possible to recognize the match in benefits for SDAIE Online as well. The seven areas where SDAIE Online 
implementation benefited participants are as follows: a) Learners experience components of low-risk and 
safe environment, b) Cross-cultural adaptations of course content that are appropriate for the target audience, 
c) Multiple representations of types of guides and learning companions, d) Components of responsive 
scaffolding in the course, e) Components conducive to low affect and less stress, f) components of pre- 
course tutorials and overviews, g) International students can understand and access academic content area. 
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articipants were required to reflect upon which of Bennett’s six stages they probably were 
experiencing. They identified the benefits of their moments of transformation in adjusting to living and 
working in a second culture. Their new views and feelings documented some positive changes from where 
they had started in the course. One major issue was related to the influence of high-and low-context 
communication and worldviews in their lives. I documented a collective sigh of relief and new views that 
participants expressed after they recognized elements of this almost invisible reality. Once the learners 
realized their high-context views and expectations were causing frustration in low-context situations at work 
and in class, they found ways to be flexible and adjust more to bicultural views. 

By the end of the implementation phase every participant indicated some levels of adjustments had 
occurred for him or her. All this change contributed to the sense of confidence and setting the learners’ 
thinking free as evidenced by the following comments by a Mexican male during our final interview: 

I think the most important thing I got from the entire course was to find out how I’m doing in some 
aspects of my culture bump and about high-low context. I found out I’m conservative in my 
speaking. Sometimes I don’t say much because I’m kind of shy. Sometimes we don’t notice it. We 
just keep leaving it like that. This class opened my eyes and let me know I need to work on that if 1 
want success here in this country. I got A LOT of information on the areas where I need to 
improve, and I got information HOW to improve. I decided I need to work harder to improve my 
pronunciation and make my accent less noticeable. 

New things for me are the stages of adjusting to cultural differences. The questionnaires help me 
more know more about myself, and this gives me ideas for improving. The information in this 
course is helping me at work. I realized that I am in three cultures and adjusting. I learn a lot about 
the U.S. culture from my co-workers, but I interact with my wife more than I do my co-workers. I 
think my wife’s American culture is different because she is from a different culture. She speaks 
normal English, but she speaks on the phone different to her mom than she does to me. Her 
pronunciation is different then because she speaks Ebonics with her grandmother and mother. I 
don’t understand it. The situation changes how I have to adapt and respond. It’s kinda complex. 

Conclusions 

One major design objective was met in this research: SDAIE Online was instantiated, and it exists 
as a design instance. This exploratory design study research allowed for some useful qualitative data to be 
gathered; however, the sample size was small. I suggest that we can assume these results will be limited to 
Spanish and Portuguese speakers who live and work in the U.S. and that the results will work for those in 
this group who expect to live abroad here for a few years or more. Nevertheless, the seven participants’ 
favorable responses to the SDAIE methods and strategies suggest that yes these learners received similar or 
some of the same benefits while participating in this SDAIE Online design instance as they would in a 
SDAIE F2F course. Their confident responses stands a proof of concept that SDAIE F2F methods can 
function in an online environment with positive results. I suggest that we can assume these results will be 
limited to Spanish and Portuguese speakers who live and work in the U.S. and that the results will work for 
those in this group who expect to live abroad here for a few years or more. 

As evaluated in the formative evaluation, the course design worked well, and I have decided that I 
can continue to use these course pieces in future course versions. I have concluded that the participants all 
could do certain beneficial tasks and recognized “invisible realities’’ that they could not before the course. 
Learners did move across cultural learning boundaries. From self-reports during our final interview, learners 
marked their spot on the continuum of the six stages of cultural differences. They have felt they had 
definitely moved on beyond ethnocentrism to some point in other stages. Connected to these changes, they 
all selected real world problems to explore in their cultural bumps and with learning strategies. From the 
data and the findings these seven students learned new concepts from course content in a safe environment. 
They were able to take action in their personal lives, reporting enhanced confidence, a positive attitude, and 
improved skills in their personal and professional lives. 

It takes a lifetime of effort and noticing subtleties to better act and to communicate in diverse 
situations. Because culture is always changing and human nature brings changes and surprises, it is fleeting 
to feel that we “have arrived”. The development of cultural effectiveness is a life long process, should we 
accept this view. I believe that no one “makes it” alone or is successful alone. No matter what age or level of 
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professional expertise we attain, we can all benefit from guides, role models, and learning companions. The 
cross-cultural nature of many of our courses and our efforts in global communication requires special 
attention because of the unintentional misconceptions in lessons that can occur during design, development, 
and implementation. I encourage others to identify and work through cultural bumps or disconnects as they 
identify the stages of intercultural competence in which they operate. As we become the leaders of the 
globally connected 21st Century world, may we become more elegant and effective in the way we approach 
our communication with others. Both leaders and the learners can be enriched by increasing their reflection 
time and having courage to explore other cultural views and experiences. 

Future Research 

This course and research are a beginning point in examining how SDAIE can be implemented 
online. By creating a prototype example of an online course and using it with multinational students in the 
USA, the design study for this research project may serve as a guide to others who look for ways to adapt 
design to more contexts and for multiple cultures. This research study may make it possible for SDAIE 
Online to be implemented into more widespread use by other designers as well. I will create more videos 
with various views all dealing with the same event and will expand on research with cultural bump 
experiences from different points of view. After refining this course and offering it commercially from an 
online hosting service, I will create more related courses and encourage others to create similar courses on 
various topics and skills. I plan to use more of the multiple cultures model (MCM) expanded by McLoughlin 
(1999) and Henderson (2007), a cultural contextualization of instructional design. In this model designers 
create resources for students to learn multiple formats that show more than one cultural viewpoint. Closely 
related to SDAIE, designers can use the MCM to adjust content and activities for learners who represent 
various cultures and view points that vary from the mainstream. “However, recognizing that instructional 
design cannot be culturally neutral is a first step in the process of becoming more culturally competent” 
(Palloff & Pratt, 2003, p. 41). 

The SDAIE Online course of this research project for Hispanics and Latinos is the foundation 
course on a series of courses for multinational learners. I am planning to create similar courses for Koreans, 
Japanese, and Pacific Islanders, for example, on topics of intercultural communication and cultural 
awareness while adapting to life in a second culture. I hope this solution of using SDAIE as a conceptual 
tool for online course design will not only reduce the number of e-learning dropouts but also increase the 
number of students who learn to transition safely into other people’s worldviews with less stress and 
confusion. A new journey of transformation awaits us all. 
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Characteristics of Job Corps Students Revisited 

Denise Tolbert, Ph.D. 



The Job Corps Program 

In 2001 a study was conducted on the Job Corps program. The Job Corps Program was established by the 
Economic Opportunity Act of 1964 and is the major youth training and employment program authorized by the Job 
Training and Partnership Act (JTPA). The program is targeted at youth who are economically and educationally 
disadvantaged, between ages 16 and 24 with the intention of breaking the cycle of poverty and improving the 
economic prospects of the targeted group. The Job Corps training centers offer basic education training leading to a 
high school diploma or GED and vocational skills training in over 100 career areas. In addition to the skills training, 
many other benefits are offered. These other benefits include: work experience, counseling, health care and other 
support services, a small monthly living allowance, a small clothing allowance, an allowance for child support, 
transportation allowance, and a readjustment fund to assist in the transition to work after the program is completed. 
With the exception of a few sites all participants reside in dormitory settings at the training sites. Program enrollees 
range in age from 16 to 24. 

Background 

There are many programs government and private, national and local, that provide academic and vocational 
training. Effective means for training low-literate, low-skilled groups to enter the workforce, at even baseline levels, 
need to be identified so that training programs can be structured to ensure their success. The Job Corps program has 
a 40-year history of training this population. 

The original study completed in 2000 consisted of all students who enrolled in Job Corps training programs 
nationally from July 1, 1996 through June 30, 1997 and investigated 5 independent variables: length of time out of 
school, age, gender, ethnicity and scores on the Tests of Adult Basic Education in English and math and their 
relationship to the dependent variables program completion and job placement. This current study re-examines the 
research questions from the original study. 

The research questions from the original study are: 

1. Does length of time out of school, age, gender, ethnicity and entry-level academic skills as measured by scores on 

the Tests of Adult Basic Education (TABE) in English and math discriminate between Job Corps students 
who complete training and Job Corps students who do not complete training? 

2. Do length of time out of school, age, gender, ethnicity and entry-level academic skills as measured by scores on 

the Tests of Adult Basic Education (TABE) in both English and math discriminate between Job Corps 
students who are placed in jobs and Job Corps students who are not placed in jobs? 

3. Is there a relationship between length of time out of school, age, gender, ethnicity and entry-level academic skills 

as measured by scores on the Tests of Adult Basic Education (TABE) in both English and math for Job 
Corps students who differ on training completion time? 

4. Is there a relationship between the Tests of Adult Basic Education (TABE) English scores and training 

completion, job placement, and length of time to complete training? 

5. Is there a relationship between the Tests of Adult Basic Education (TABE) math scores and training completion, 

job placement, and length of time to complete training? 

The current study consists of all students who enrolled from July 1, 1997 to June 30, 2006 and examines 
the same research questions. The first two hypotheses were tested using discriminant analysis; hypotheses 3, 4 and 5 
were tested using multiple linear regression. 

Results 

In the original study 48.8% of enrollees completed training and 85% were placed in jobs. An examination 
of training completion rates showed enrollees 18 years or younger had the lowest rate at 44.83%. The rate rose 
through the age groups to a high of 56.98% for enrollees ages 21-24. Training completion time was about one year 
for 98% of enrollees. Job placement rates for enrollees age 18 and younger were 69.3%; this rate rose to 74.8% for 
enrollees age 21-24. Performance by gender showed 49.9% of females completed training and 70.7% of them were 
placed in jobs. Men performed equally with 48% completing training and 71.7% were placed in jobs. Data showed 
that older students have a higher success level than their younger counterparts. 
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In 2000 training completion rates by ethnicity were: Asian Pacific Islanders 63.8%, Hispanics 51.2%; 
White/Caucasian 49.5%; Black 47.2%; and American Native/Alaskan Native 45.6%. Job Placement for these groups 
was: Asian Pacific Islanders 75.6%; White/Caucasian 75.5%; Hispanic 73.5%; Black 68.6%; and American 
Native/Alaskan Native 64.3%. Among the ethnic groups Asian Pacific Islanders had the highest performance rates 
in both training completion and job placement while American Native/Alaskan Native had the lowest. Hispanics, 
White/Caucasian and Blacks have quite similar performance levels. 

One of the characteristics of at-risk learners is low literacy levels. This is borne out in the Test of Adult 
Basic Education (TABE) English results of enrollees. Scores of 500 or less equate to approximately a 1 st grade 
literacy level. In the original study training completion rates by TABE scores were: 43.6% of enrollees who scored 
500 or less; 50.5% of those who scored 500-600; and 61.6% of those who scored more than 600. Job placement rates 
were: 66.4% for those who scored 500 or less; 73% for those who scored 500-600; and 78.2% for those who scored 
more than 600. Enrollees also take a TABE in math. The training completion rates were: 43.8% of those who scored 
500 or less; 53.7% of those who scored 500-600; and 65.6% for those who scored more than 600. Job placement 
rates were: 67% for those who scored 500 or less; 53.7% for those who scored 500-600; and 65.6% for those who 
scored more than 600. Despite low literacy levels, enrollees have favorable training completion and job placement 
rates. 

Data categories in the most recent study do not mirror the original study. This is due to two factors: 
different data recording policies used by the Department of Labor and use of different software (SPSS vs. SAS) for 
analysis. In the 2000 study training completion and job placement were recorded as yes or no. In the current study 
these variables are reported by gender, age and ethnicity. 

Training completion data by gender reveals the following: females 59.4%, males 57.4%. Breakdowns for 
completion by age are: 16 years or younger, 50.0%; 17-19 years, 57.5%; 20 years or older, 65.0%. Training 
completions by ethnicity are: American Indian or Alaskan Native, 55.3%; Asian/Native Hawaiian/Pacific Islander, 
67.8%, Black or African American, 58.1%; Hispanic or Latino, 57.8; White/Caucasian, 58.5%. Results indicate 
differences in training completion by gender are not significant. Older students complete training at higher rates than 
their younger counterparts. 

Further job placement data is divided between those placed in the area of vocational training or not. The 
percentage of females placed in jobs in the area of their vocational training is 26.2%; those placed but not in the area 
of vocational training is 54.1%. The percentage of men placed in jobs in the area of their vocational training is 
26.8%; those placed but not in the area of vocational training is 44.8%. By age job placement in the area of 
vocational training is: enrollees 16 years or younger, 17.8%; 17-19 years, 26.4%; 20 years or more, 32.3%. 
Percentages of enrollees placed outside their area of vocational training are: 16 years or younger, 53.5%; 17-19 
years, 48.2; 20 years or more, 46.0%. 

Job placement by ethnicity in the area of vocational training is: American Indian or Alaskan Native, 24.6%; 
Asian/Native Hawaiian/Pacific Islander, 31.3%; Black or African American, 24.1%, Hispanic or Latino, 27.1%; 
White/Caucasian, 30.4%. Job placement outside the area of vocational training is: American Indian or Alaskan 
Native, 48.5%; Asian/Native Hawaiian/Pacific Islander, 50.4%; Black or African American, 50.1%; Hispanic or 
Latino, 50.8%; White/Caucasian, 44.2%. For all categories, gender, age and ethnicity, fewer than half are placed in 
jobs in the area of vocational training. These figures may be low due to the percentage of “Unknown,” those whose 
records were incomplete. Overall 96.1% of enrollees completed training in 12-18 months. 

Low literacy levels continue to be a characteristic of this group. On the TABE English test 50.7% of 
training completers scored less than 500; 19.8% scored 500-600; 68.7% scored 600 or more. On the TABE math test 
51.5% scored less than 500; 61.3% scored 500-600; 70.9% scored 600 or more. Data for TABE scores by job 
placement were not reported. Literacy levels have not changed significantly between the two studies. Despite these 
low levels enrollees are still completing training at fairly high rates. 

These results show that the Job Corps continues to be an effective program for at-risk students. Despite 
entering the program with low literacy levels students are able to complete training programs that lead to 
employment in vocational areas. 
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Open Source Technology Solutions: Free Alternatives to Expensive 

Commercial Products 



Eddie Vega 

Department of Adolescence Education / Health 

introduction 

The term “open-source” is synonymous with words such as, “free” and “Linux”. True, “open-source” is 
linux-based. And “yes”, it is free. Linux-based products are designed originally for people who understand the 
concept of what makes Linux work. However, the producers of these products also want to gain momentum by 
providing both Windows and Mac versions of their “freeware” in hopes of making the public-at-large aware of 
alternatives to mainstream commercial applications. 

Open-source has come a long way. Many of todays open-source operating systems provide technical 
support in some capacity, open-source applications such as the alternative to Adobe Photoshop entitled, GIMP 
(GNU Image Manipulation Program) are supported by the open-source community. Therefore, when one looks up 
tutorials on how to use a specific application, most times one will find a long list of individualized sites that can 
satisfy a variety of needs. 

Though the focus of this paper is to shed some light on open-source applications, one cannot do so 
properly without a small mention of the Linux operating system. 

“Hi, I'm a Mac! I'm a PC! And I'm Linux?” - Some Linux Information 

The names Bill Gates and Steve Jobs can be associated to Microsoft and Macintosh respectively, but who 
is Linus Torvalds? Originally from Finland, this graduate from the University of Helsinki, who now calls Silicon 
Valley home, is considered to be the father of Linux and a hero to many. 

Linux, like Windows and Mac, is an operating system, or “o/s”. An operating system establishes an 
environment where one can have access to a multitude of tools in order to create/produce a product (i.e. : 
spreadsheets, presentations, word processed documents, graphics, check emails, listen to music and web designs, 
just to name a few). Linux, unlike the mainstream giants, has a variety of operating systems to meet the needs of 
individuals with varying skill levels. Conversely, Windows provides users with several versions of its o/s with 
varying degrees of functionality and Mac provide its users with only one option in an operating system. 

Some factors that set these operating systems apart are: l)Linux is FREE, whereas Windows and Mac 
carry hefty price tags. 2) The code behind Linux itself is “open-source” and therefore can be modified and re- 
written (the reason behind why there are, quite literally, hundreds of various versions of Linux - each sporting a 
different name, look and feel). 3) A programmer with enough skill to redesign the internal coding of the operating 
system, which is open-source - meaning, no one individual owns it - can then turn around and make a profit. 
Companies such as Red Hat and Novell are currently making a profit selling their versions of the o/s as well as 
technical support. 4) Additional applications, such as browsers, office suites, games, graphic design software, etc., 
can easily be allocated throughout the open-source community. On the commercial side, there is a price tag 
associated to every piece of software we use. However since Linux, by design is free, it is referred or 

associated to the world of open-source. 

A full timeline of the History of Linux can be found online at: http://www.linux.org/info/linux timeline.html 

A brief biography for Linus Torvalds can also be found online at: 
http://www.linux.org/info/linus.html 
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Demystification 



When one hears the phrase “open-source” or the word “Linux”, the immediate reactions are that it sounds 
complicated or that there's just too much programming involved. The purest form of Linux does require a solid 
understanding of computer programming. One of the reasons for this is that by using what is known as typed 
“command lines,” a user can gain direct access to a particular application without having to wait to locate the 
application, then for the “pretty” graphics to reveal the application, and finally for the application to open up in 
order to work. Although the ease of Linux and command lines may sound efficient, they pose certain challenges for 
individuals who want significantly more than what other operating systems can offer. For many people who are 
accustomed to working in a Windows or Mac environment, working in a graphic interface with point-and-click 
features makes the experience a sane one. The idea of having to type a command that will open some program 
similar to Word doesn't thrill many users. And, as if that weren't enough, a user would need to troubleshoot his/her 
command if an error is encountered. 

The graphic interface, or Graphical User Interface (GUI), is vital in winning over users of traditional 
operating systems. Currently, this has become an explosive topic, but in a good way. More and more Linux-based 
operating systems now have a GUI for those who want to try Linux without the fear of the “command line.” Such 
operating systems include, Mandriva, Knoppix, Fedora and the Ubuntu Family - Ubuntu, Kubuntu, Edubuntu and 
Xubuntu. These o/s allow users to simply point-and-click to access any application. (And “yes”, there is the right- 
click for those Windows fans.) 

Running a Linux o/s requires you to understand that the commercial or proprietary applications that would 
normally run on your Windows or Mac will not always run on Linux. (There is a way to do this for Windows 
applications, but details will not be given here simply because it is not foolproof and may cause more harm than 
good.) Perhaps more importantly, knowing what the alternative applications are to the commercial and proprietary 
products will help users to achieve a certain level of financial freedom. 

Open-Source and Financial Freedom 

As stated earlier, open-source applications are “freeware.” These programs are produced and maintained 
by people from all over the world who believe that no matter what an individual's economic status is, what country 
someone lives in, what level of education s/he may have or may not have - everyone is entitled to have access to 
technology. As a result, open-source is not solely 100% for Linux. There are open-source versions of applications 
available to both Windows and Mac. 

The equivalent to Microsoft Office (MS) is called, Open Office (OO). It is available for Linux, Windows 
and Mac. It's a free download from www.openoffice.org . Simply select the version for the appropriate o/s and 
download. Open Office contains equivalent applications to rival Microsoft. For example, OO Writer is MS Word; 
OO Impress is MS PowerPoint; OO Calc is MS Excel; OO Base is MS Access, and so on. Advantages to Open 
Office include: Microsoft Office compatibility; export to PDF almost instantly; creation of a SWF (ShockWave 
File) of a Presentation; files saved to earlier versions of Microsoft; files saved to WordPerfect; and many more. 
This article was created using Open Office Writer. Can any noticeable difference between this and a Microsoft 
Word document be detected? 

For graphic design, the most popular commercial product on the is Adobe Photoshop. Version 10 (CS3), 
as of this writing, is now on the market. Adobe is now releasing new versions on a yearly basis. It is nearly 
impossible for anyone, organization, business or individual, to keep up with this technology in a cost effective 
manner. It even that much more difficult for non-profit organizations, small businesses, educational institutions, or 
students who either operate on a limited shoe-string budget or no budget at all to remain on the cutting edge of 
software like this. However, there is an open-source Photoshop equivalent called the GNU Image Manipulation 
Program, better known as GIMP - www.gimp.org . Like Open Office, it is available for Linux, Windows and Mac. 
At time of this article, GIMP is at version 2.2 and is about as powerful as Photoshop version 7. It has many of the 
same features, but like all new programs, it does take some time to get oriented to its interface. 
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If a user needed to build a website and could not find a current version of Microsoft FrontPage (Microsoft 
officially dropped FrontPage as of November 2006) and Adobe Dreamweaver is not within budget, NVU is an 
open-source option - see www.nvu.org . There is also KompoZer - www.kompozer. org - which is built on the NVU 
interface and the programming code used to design the application itself. As with KompoZer, an advantage of 
open-source is that no one individual or organization owns the code behind the application. It can be modified and 
renamed and put back out onto the web as a new product or as an upgrade. Another advantage is that open-source 
applications are available in many languages. NVU or KompoZer is a WYSIWYG (What You See is What You 
Get) text editor as powerful as FrontPage 2000. 

Overall, what would a package containing the latest Microsoft Office Suite, Adobe Photoshop and Adobe 
Dreamweaver cost? Walk into any computer or office supply store and the prices range from the ludicrous to the 
absurd. For teachers, faculty, staff, and students, there are academic discounts from websites such as: 
www.academicsuperstore.com , w ww. i ourneved. com , www.studica.com. just to name a few, but the problems begin 
when an institution or organization must go with the lowest bid on software from specific sites yet those prices are 
still over-the-top. To illustrate the open-source versus commercial costs of equivalent products, see the chart 
entitled “Cost of Commercial Applications vs. open-source Applications.” 

Whether an individual runs a P-12 classroom, or a corporate boardroom, open-source is making its way 
into the mainstream - and saving users money. Classrooms can now conduct courses that will keep students on the 
cutting edge. Companies can compete with the ever-changing digital techscape by providing high-caliber products 
and still remain within or below budget costs. Funds that would usually find themselves being spent on software 
can be utilized in other areas of an organization's infrastructure. Hence, financial freedom. 

In this day and age, there comes a time when there will be a need to re-structure organizations, or downsize 
a company or make other sacrifices based on economic snapshots and projections. When the time comes to make 
decisions on upgrading existing software, the technology-using population raises a collective eyebrow and scratches 
their heads. Open-source can help during the transition period between software upgrades or may even substitute 
for the upgrade and become the permanent application of choice. 

Where does Open-Source Technology fit in? 

Name an industry and a need exists for open-source to fill. Regardless of budget limitations, open-source 
finds its market. In Hollywood, films such as Eragon, Ghost Rider, Spider-Man 3, the Pirates of the Caribbean 
trilogy and even the upcoming Transformers and Fantastic Four features have used 3D software to create some of 
the best larger-than-life scenes ever to fill the big screen. The video game industry is currently one of the industries 
that is also experiencing a boom, with 3D graphics bringing a new level of realism to the small screen. So, where 
do designers and animators go to learn to use 3D technology? Well, schools, colleges, and professional 
development workshops to name a few, but these can be costly for both the student as well as the training facility. 
Open-source solutions can help. For instance, while Maya, an industry leading 3D animation software from 
AutoDesk ( http ://usa, autodesk.com/adsk/servlet/index? siteID= 12311 2&id=763 50 18 ), is by far the best and the 
most expensive - the cost is in the ballpark of $7,000.00, there are open-source alternatives. Enter - Blender! A 
powerful cross-platform suite of tools that allows for the creation of 3D content and offers playback. This means 
that there is a version for Windows, Mac and Linux. The best part of this is that it is all free to download from their 
website (see Blender - http://www.blender.org/ ). Even Hollywood films with their stellar-size budgets use open- 
source technology. In the case of Spider-Man 2, Blender was used to set-up the visuals (see 
http://www.blender.org/features-gallerv/testimonials/animatics-for-motion-pictures/ ). Now more than ever do the 
industries around the world need talented individuals with 3D skills. Whether in films, video games, engineering, 
medical fields, corporate training, architecture, manufacturing, etc., staying on the cutting edge doesn't mean that 
current or up-and-coming technology professionals need to spend a lot of money on software products. 

Problems Facing Academic Software 

Academic software, because of its nature, is illegal to use outside of the educational environment. This 
software was merely intended to be used within the classroom setting. Although this software may not have any 
restrictions on what its capabilities are, the restrictions lie in how this software is used in producing content and for 
whom. If a programmer wanted to view the coding behind a piece of work, s/he may find the words “Academic 
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Version” embedded throughout the code. This can lead to some serious legal issues. Again, the solution, to this 
dilemma is open-source products. 



The World at Large: Who uses Open-Source? 

The industries are finding themselves facing down similar situations to the above statements. In an article 
published on the “Linux Insider” website, the Federal Aviation Administration (FAA) is strongly considering 
switching to Linux and open-source technology rather than continue with Windows-based system. 

(Original article: “FAA May Choose Linux over Vista” - http://www.linuxinsider.com/storv/56196.html ) 

(Similar story: “Can Google and Linux Topple Microsoft” - 

http://technewsworld.com/story/Xi5rS5HtkYtAtW/Can-Google-and-Linux-Topple-Microsoft.xhtml ) 

CEO Michel Dell of Dell computers, uses the Linux o/s, “Ubuntu” and is currently selling pre-installed 
Linux on certain model desktop and laptops. (Press Release from Dell: “Dell Unveils Three Consumer Systems 
Featuring Ubuntu 7.04”- 

http://www.dell.com/content/topics/global.aspx/corp/pressoffice/en/2007/20Q7 05 24 rr OOQ?c=us&l=en&s=corp ) 

Pushing the boundaries of open-source technology, an independently produced animated film was created 
entirely in Blender, entitled, “Elephants Dream”. To download the [815mb] movie, visit, “Elephants Dream” 
website at: http://www.elephantsdream.org/ For a list of the open-source technology used in the production of this 
movie, please visit, http://en.wikipedia.org/wiki/Elephants dream These are just a few of the many cases studies 
surrounding open-source and various industries. 



Summary 

Although global industries push for the latest technology, even they occasionally feel the noose around 
their budgets. Open-source is an alternative to the high-priced applications being put out there by manufacturing 
giants. Linux is open-source, but open-source can be utilized on existing Windows and Macs. With vast open- 
source resources, it is now very possible for anyone trying to establish themselves in the digital world to do so 
without “breaking the bank”. Open-source is here. It's cross-platform. And most importantly, it's FREE! 
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Conversation analysis as a framework to design and to evaluate 
Computer-Supported Cooperative Learning (CSCL) Environments 



Patricia Verdines 
Tecnologico de Monterrey 

This paper describes a qualitative study conducted to understand the communication behaviors and the instructional interactions among 
instructors, learners, and content in CSCL environments. It also describes the extent to which the systematic analysis of online 
conversations can provide evidence of learning processes and learning outcomes in those environments. 

Introduction 

Early research on instructional communication characterized teachers as a senders; the students as receivers; the course content as the 
message, and the instructional media as communication channels (Heinich, Molenda & Russell , 1996). This study builds upon previous 
research by characterizing online conversations as instances of communication events that converge to influence and shape the 
instructional communication process in CSCL environments. 

In the study, the target audience is composed of adult learners, and the learning paradigm within which the conversations were analyzed is 
constructivism. The constructs and research questions guiding the analysis were grounded in learning theory, instructional design theory, 
and the theories represented by various communication models. The communication behavior and interactions among participants in a 
distance education course with a computer-supported cooperative learning (CSCL) environment were analyzed at a “macro” and a 
“micro” levels. 



Research questions 

The research questions guiding the study are: What is the nature of the instructional communication process in constructivist distance 
education environments? How is it supported by computer-mediated instructional systems for adult learners? 

Procedures 

Unit of analysis: A course was selected following a theoretical construct sampling strategy. The selected course was designed and 
delivered online with WebCT, which is a virtual course environment with both synchronous and asynchronous communication tools. It 
provided access to the course archival data with the online discussions and conversations among the course participants. 

Data collection: The various kinds of online messages created by the instructional team, guest speakers and students in both synchronous 
and asynchronous conversations were conceptualized as documents representing instructional interactions and communication events. 
Relevant information within each of the course modules was selected and transcribed according to a purposive sampling strategy (Ten 
Have, 1999) to obtain a wide range of types of data. To protect the identity of all the course participants, their names and affiliations were 
edited within all transcripts. 

Data analysis: The procedures for data analysis in the study include: (a) conversation analysis (Ten Have, 1999) to explore the nature of 
the instructional communication process in a CSCL environment and (b) content analysis (Weber, 1990) to identify the types of 
interactions, cognitive processes and types of knowledge represented by the participants’ discourse. The procedures for conversation 
analysis attempted to answer foreshadowing questions at the “macro” level, while the procedures for content analysis addressed questions 
at the “micro” level. To increase the credibility of the research results, two other researchers were asked to analyze and code a sample of 
online conversations for comparative purposes. Then, a member of the instructional team who actively participated in the course was 
interviewed to triangulate her perspectives with the researcher’s interpretations. 

Findings 

The conversations in the data set were characterized as instances of instructional communication events with learning goals and specific 
rules guiding the participants’ interactions and communication behavior. Despite the differences in terms of the instructional settings and 
the intended audience, the actions and discourse sequences identified in the literature as a basic framework for the analysis of face-to-face 
classroom discourse (Cazden, 2001; Mehan, 1985) were easily adapted and used to guide the analysis of online conversations in CSCL 
environments. 

In the study, a single conversation had more than one discourse sequences; a discourse sequence included more than one turn by each 
participant; and a single turn represented one or more actions by each participant. Synchronous conversations did not promote interactions 
among the students; however, they provided diverse opportunities for the students to extend their understanding and to share their 
knowledge by interacting with several guest speakers’ who participated as experts in the domain. The participants’ discourse in 
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asynchronous conversations represented much more diverse interactions among instructors and students, including interaction initiated by 
the students. 

The students’ discourse provided extensive evidence of their ability to understand, apply, analyze, and evaluate the information delivered 
as course content by the instructional team, which is consistent with the cognitive processes involved in the course objectives as learning 
goals. Several types of knowledge were also represented within the students’ discourse in the data set; however, not all types of 
knowledge were represented in all conversations nor were all types of knowledge constructed and shared by all students. 

The students’ discourse was characterized as evidence of (a) collaborative learning processes in which the students shared their individual 
experiences to solve a problem or to accomplish a given task without the supervision of any instructor, or (b) cooperative learning 
processes in which the students followed specific guidelines from the instructors in forming teams to accomplish tasks defined by the 
instructors as well. 



Implications 

A better understanding of the nature of the instructional communication process in CSCL environments could serve as an opportunity to 
revise and refine current instructional design practices for those environments. For instance, the analysis of online conversations as 
cooperative teaching/leaming opportunities could be part of a formative evaluation protocol with a focus on identifying the conversational 
features that converge in distance education environments. The protocol could also include the analysis of online conversations as a 
procedure for summative evaluation to identify the roles and the types of knowledge represented in the discourse of individual students in 
the course. 
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F 1 » OK, we are ready to get started. Today we welcome E2 . . . O: FI opens a synchronous conversation in a chat room session 
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Appendix A: Conversation analysis of a discourse sequence from a synchronous conversation 
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Appendix B: Content analysis of a discourse sequence from a synchronous conversation 



Educating Educational Designers: The University Of Twente Case 



Irene Visscher-Voerman 
Wilmad Kuiper 
Pl0n Verhagen 

Summary 

How do you efficiently and effectively educate students to be academically competent and practice-oriented 
educational designers when they lack prior design experience? This papers provides an answer to this question by 
describing and discussing a four-year University program in instructional design and educational technology. The 
program incorporates substantive developments in the field, and uses authentic and partly technology-based 
approaches to teaching and learning educational design. 

Introduction And Problem Statement 

The basic curriculum question ‘what knowledge is worth teaching’ is very actual for those engaged in 
teaching designers. While we are changing our views on how designers work in practice, we need to reconsider the 
basic design models and design knowledge we are teaching (novice) designers. And there is also good reason to 
question whether the traditional teaching pedagogies are still sufficient to introduce novices to the field. Despite 
some good examples or ideas for the education of designers (c.f. Dorst & Reymen, 2004; Rieber, 2000; Rieber, 
Orey, & King, 2006; Schon, 1987; Rowland, Fixl & Young, 1992; Shambaugh & Magliaro, 2001; Visscher- 
Voerman, 1999) the attention for this topic lags behind the attention for what designers actually do. As a first and 
important step towards further increasing the quality of design education, it is important for educators to share their 
assumptions, theories, and experiences with teaching designers and to provide their answers to the afore-raised 
question. It is the intention of the authors of this article to share their assumptions, views and experiences with 
teaching (novice) designers in their program, as it may inspire other design educators. 

The authors work as teachers and managers in a four-year educational design program at the University of 
Twente in the Netherlands, which recently celebrated its 26th anniversary. An important feature of this program is 
its multidisciplinary educational design orientation. This orientation was inspired by the North American tradition in 
the field of instructional design and educational technology, a field that encompasses the systematic analysis of 
learning and performance problems, and the systematic design, development, implementation, evaluation, and 
management of instructional and non-instructional processes to improve learning and performance in a variety of 
settings (Reiser & Dempsey, 2007). 

Whereas there is overlap with a lot of design programs all over the world, this program has three distinct 
features. First, it is the only four-year design program in the Netherlands, with a 100% focus on educational science 
and technology. It consists of an undergraduate level (first three years of the program leading to a bachelors degree) 
and a graduate level (last year of the program leading to a masters degree). Whereas most students enrolling the 
undergraduate level are novices to the field of educational design, the graduate level designates students with a basic 
level of design knowledge or experience. 

A second major feature of the program is its aim to educate students to be able to work in both academic 
settings, e.g. as design researchers, and professional contexts, e.g. as designers or as consultants. 

Third, the program has a broad focus, providing theory, design perspectives, and research approaches in 
several educational domains (curriculum, learning and instruction, media, educational organization and 
management, educational measurement, and human resource development). Because of this broad focus, the term 
‘educational design’ has been used rather than ‘instructional design’, as the latter evokes too much of an association 
with business environments (Richey, Fields, & Foxon, 2001; see also van den Akker & Kuiper, 2007). 

As the basis of this program are four major design principles, e.g.: 

• Design principle 1 : Initiation into the academic profession as guiding pedagogical approach 

• Design principle 2: The instrumental approach as leading pedagogical device, but with sufficient attention to 
various other main design approaches 

• Design principle 3 : Explicit coupling between generic and domain-specific design approaches and principles 

• Design principle 4: Explicit coupling of educational design and research 
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In this article, the authors will first describe the program as a kind of advanced organizer. Secondly, they 
will articulate the rationale and indicate why and how these design principles permeate the program, embedding 
them in both developments and trends in the design field and own experiences. Thirdly, the authors will provide 
evaluative data from a survey amongst alumni and an external university audit to sketch how this program is valued 
by both practitioners and academics. It is not so much meant as to proof or disproof the quality of the program, but 
to provide the reader with the complete picture, from curricular intentions to teaching practice to results. Fourthly, in 
the same line of reasoning, the authors raise several issues and bottlenecks, stemming from the program, its 
underlying rationale, and the experiences. The reader may be inspired to judge to what extent ideas and elaborations 
may be relevant for their own design teaching context. 

Description Of The Bachelors And Masters Curriculum 

The current bachelors program has evolved over 25 years. Although major elements have been there, the design 
principles have only been formulated recently. 

There have been formulated 5 competence areas for the program, which stem directly from the program’s 
focus on educational problem-solving. In order to analyze and understand educational problems students need 
theoretical knowledge (reflected in competence area 1 : domain knowledge) and research skills (reflected in 
competence area 2: research competencies); in order to solve them, they need design/development and 
research/evaluation knowledge and skills (reflected in competence area 3: design competencies) (see Figure 1; 
adapted from Verhagen, 2000). To operate within the triangle theory-design-research, students require social, 
communicative, information technology, and reflexive skills (reflected in competence area 4: general academic and 
professional skills). Since many designers come to work as consultants in either the design or research area, the 
students also need to master advisory competencies (competence area 5). 

Figure 1. Educational problem solving 
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These competencies apply to both the three-year bachelors program and the one-year masters program. The only 
difference is that in the masters program the competencies are related to one specific domain and students are 
expected to demonstrate a higher level of mastery, as formulated in the European ‘Dublin descriptors’, with respect 
to ‘knowledge and understanding’; ‘applying knowledge and understanding’, ‘making judgments’; 
‘communication’; and ‘learning skills’ (see Joint Quality Initiative, 2004). All competence levels get equal attention 
in the programs, except for the advisory competencies which are devoted less time. 

Structure Of The Bachelors Curriculum 

The Educational Design Management & Media (EDMM) program consists of four main components, in 
which several courses are clustered. In Figure 5 the structure is visualized. This is not a chronological structure, but 
a content-related structure where the arrows show which courses are related. The student study load of each course 
is being expressed in terms of European Credits (EC), with one EC equaling 28 hours of total student work. This 
may include attending classes, reading, group work, working on assignments, writing reports, and meeting teachers 
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for feedback depending on the amount of independence expected from students. As such, each academic year 
matches 60 European credits; the full bachelors program consists of 180 EC. 

The first component consists of th e foundational courses. Here, an introduction is provided to the field of 
education and training; to related domains of psychology, sociology, andragogy, and pedagogy; and to one of the 
educational domains, human resource development. In these courses, students acquire basic terminology and create 
a knowledge base that they can rely on in other courses. 

Second, the design stream consists of an introduction course and five design ateliers in combination with 
nine supporting courses. The first four ateliers are linked with a specific educational domain (media; curriculum and 
instruction; organization and management and human resource development; and evaluation and assessment). The 
fifth atelier fully consists of an authentic ‘synthesis’ design project to be conducted in groups in the domain of 
choice. In the course ‘Educational design: Introduction’ an introduction is provided to design methodology in 
general and to each of the specific ADDIE-phases and activities in particular. In the ateliers, students practice design 
techniques and methods in each of the domains with increasing complexity. 

Third, students in their third year should choose several courses in a discipline different from the 
educational domain for a broader academic orientation. This set of courses is called the minor and stems from the 
University of Twente philosophy that attending courses in other domains develops and broadens students’ academic 
competencies, such as communicating with professionals from other disciplines, having a broad perspective on 
science, and being able to master new topics relatively quickly. 

Fourth, the research stream consists of six courses and, at the end of the third year, an individual research 
project. In the courses students are taught research methodology, research designs, methods of qualitative and 
predominantly quantitative data gathering, and statistical analysis techniques. The research assignment (more or less 
in a design context) requires a student to demonstrate his or her competencies as a junior researcher. 

Structure Of The Masters Curriculum 

In the masters program Educational Science and Technology, students narrow their specialization further in 
one particular educational domain. They choose one out of three different tracks: Curriculum, Instruction & Media 
Applications (CIMA), Educational Measurement, Evaluation, and Assessment (EMEA); or Organization 
Psychology and Human Resource Development (OP&HRD). The masters program is comprised of 60 EC, and can 
be attended full-time (one year of study) or part-time (two years of study). 

Each of the masters tracks consists of three different chronological components. First, the introduction 
course of 5 EC provides a domain-specific overview and specifies the content areas to be taught or competencies to 
be developed in the track. 

Second, there are two core and three elective courses of 5 EC each. The core courses explore the domain in 
more detail and relate to one or more of the general competencies. In the elective courses, students can broaden their 
knowledge and skills within either the masters track or in other tracks depending on their specific interests. 

Third, the specialization phase is comprised of 30 EC. In this phase, students conduct a review of literature 
on research or design methodology, and on the content of their final design and/or research project. The other 20 EC 
are used to conduct the final project and to write the scientific thesis. 

Design Principles Permeating The Bachelors And The Masters Curriculum 

Now that the reader has a general picture of the structure of the program, we share the design principles 
that underlie this program and that helped us to focus choices with respect to content, pedagogies, assessment, etc. 

In this section we provide some illustrations and indicate (where possible) where they are clearly linked to each 
other. 

Design principle 1: Initiation In The Academic Profession As Guiding Pedagogical Approach 

Throughout the program, students must be prepared for the roles of academically qualified professional 
designer, researcher, and advisor. They are expected to acquire theoretical knowledge, to build design/development 
and research/evaluation knowledge, and to develop the creative ability to choose and apply a combination of those 
skills to fulfill design, research, and consultancy tasks in known and unknown authentic situations (see also Figure 
1). In view of that and inspired by the Studio Experience at the University of Georgia (Rieber, 2000; Rieber, Orey, 

& King, 2006), a pedagogical approach is intended as initiation into the academic profession (Verhagen, 2000). 
Students are induced to develop their competencies through conducting authentic tasks, both individually and 
collaboratively. During all these activities, teachers assume the role of masters, showing students in words and 
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actions how academic professionals would act. Students are increasingly expected to act as independent self- 
regulating and autonomous academic professionals. Therefore, the amount of one-on-one time between students and 
teachers gradually decreases, whereas the complexity of the assignments increases. The role of the teacher slowly 
shifts from instructor to coach and master. In the third year, students work on two large assignments supervised by a 
teacher: a design project in groups and an individual research project. The combination of these two projects 
constitutes the final stage of the bachelors program. The initiative is placed with the students, but teachers are very 
approachable with flexible schedules. In this respect it is important to realize that the financial reimbursement for 
teachers is the result of the quotient of the number of EC times the number of students attending the course. It is not, 
as is the case in many countries, related to the number of formal contact hours with students. As a result, it is easier 
for teachers to develop learning activities that ask for a reasonable amount of student independence and to provide 
flexible supervision geared to the individual or group needs. This culminates in a master-apprentice approach. 
Working on authentic assignments is considered the bridge between theory and practice. Students leam design skills 
through working on authentic assignments (mostly in groups). Assignments may include a multimedia product for 
an educational purpose (Atelier 1), a web quest as an application of instructional theory (Atelier 2); a written advice 
for organizational (re)stmcturing, school policy or human resource development policy (Atelier 3); or a professional 
project proposal and offer, as well as educational materials such as lesson series on math for autistic children or an 
electronic learning environment for higher education students, (Atelier 5). In the course ‘Curriculum, Instruction and 
Media: Practical Orientation’ practice is further intensified through cooperation with a teacher college in the 
neighborhood. In all these situations, students are expected to draw on theory, which is offered in the foundational 
courses and in the nine theoretical courses within the ateliers. 

Practice comes into the program not only through authentic assignments, but also through alumni or other 
practitioners who incidentally function as guest lecturers (for instance, Human Resource Development Theory) or in 
a jury to assess student work (Atelier 3). 

The concept of master-apprentice also permeates throughout the masters, but here, contact time with 
teachers is increased since there is only one year of study. 

Design principle 2 : Attention To Various Design Approaches, With The Instrumental Approach As Leading 
Pedagogical Device 

Over years, the field has expanded to all education and training sectors manifesting itself in a rich variety of 
instructional or educational design theories and models for various problem types and multiple settings (see for 
example Andrews & Goodson, 1991; Gustafson & Branch, 2002). Some of the design models are conceptual while 
others are procedural (Richey, 2005). ‘Procedural’ models represent recommended ADDIE-based steps to follow in 
a design process. The majority of these models pertain to either large or small scale comprehensive design projects. 
They are, for the most part, derived from applications of general systems theory. Other procedural models like 
Gagne’s (1985) Events of Instruction Model, for example, address more specific aspects of the design, development, 
and evaluation processes. ‘Conceptual’ design models identify variables that impact the design process and show 
their interrelationships. An example of these types of models is the Five Domains of Instructional Technology 
Model (Seels & Richey, 1994). 

More recently, analyses of professional design practices have shown that design approaches are much less 
homogenous and much more diverse than suggested in literature, due to differences in the kinds of design products 
to be created, design contexts, and the designer’s personal preferences and amount of experience. For instance, some 
designers follow the ADDIE phases successively or linearly, while others jump back and forth between different 
phases and/or conduct activities concurrently. Part of these differences can be traced back to the basic assumptions 
of designers on what is a good design and what constitutes a good design process. Acknowledging this, Visscher- 
Voerman and Gustafson (2004) distinguish four alternative design paradigms and underlying rationalities, 
instrumental or ‘planning-by-objectives’, communicative or ‘communication to reach consensus’, pragmatic or 
‘interactive and repetitive try-out and revision’, and artistic or ‘creation of products based on connoisseurship’. In 
their study, they found designers, not only with instrumental rationality (amongst 14 interviewed professional 
designers) but also with a communicative rationality (7) and three designers with a pragmatic rationality. 

Not only in practice can be found an increase in other approached than the instrumental one, this holds also 
for academic design projects. For example, amongst academic design projects in the University of Twente, we view 
an increase in particularly the pragmatic paradigm, as an exemplification of design research (see below) (e.g. 
McKenney, 2001; Nieveen, 1997; Keursten, 1994; Roes, 1997; Thijs, 1999; van den Berg;1996; Visser, 1998; 

Voogt, 1993). 

The vast expansion of design theories, models, and approaches made us realize that these different 
approaches should be addressed in the program to some extent. At the same time, it has been our experience that 
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students, being novice designers without any experience, profit most from learning to apply a sequential and 
structured problem-solving design approach. Therefore, it is considered important to use the instrumental approach 
as a basic ‘pedagogical tool’. This means that students should first become familiar with the ins and outs, in theory 
and in practice, with this approach before moving to other approaches. The instrumental way of designing, provides 
novices with clear guidelines and steps for all kinds of design activities that could be conducted in a variety of 
design processes and contexts (Visscher-Voerman, 1999). It can also serve as a baseline against which theoretically 
different and innovative notions can be discussed (Self, 1997). 

At the same time, however, students need to learn about and experience other less structured design 
approaches, such as the communicative, pragmatic, and artistic approach, to develop from a novice to a more 
experienced designer. During the program, they need to develop sensitivity to different design process options and 
become more flexible in choosing one approach. In the courses Educational design: an Introduction, the four 
paradigms are introduced to the students, with an emphasis on the instrumental approach. Atelier 1 asks students to 
apply the instrumental approach, atelier 2 guides students through a pragmatic approach, atelier 3 leans towards the 
communicative approach, atelier 4, again, is instrumental. In atelier 5 students are expected to deliberately choose 
and justify a design approach, and then reflect on its merit. However, it is our experience that students primarily and 
rather spontaneously apply an instrumental problem-solving approach even at the end of the program. In general, 
part-time students with more practical (design) experience are more able to value the different design approaches 
and put elements of those approaches into practice. In the masters program, the generic model is also followed, but 
tailored to the more domain-specific design model from the respective masters track (see also next design principle). 

Design principle 3: Explicit Coupling Between Generic And Domain-specific Design Approaches And Principles 

Studies of design practice have shown that design processes are highly influenced by various factors in the 
design context, such as the type of problem to be solved and the type of intervention to be designed. These factors 
have been incorporated into design-specific models (some of which have been mentioned above), and as such those 
models are quite helpful in addressing domain-specific accents, interests, needs, and wishes regarding design 
approach and interventions. Design models in the domain of educational media, for instance, provide clear 
guidelines for user-interface design or design of web-sites, and in doing so create a preference for pragmatic 
approaches. While facing the ultimate challenge of bridging the gap between policy intentions, classroom practices 
and student gains, curriculum developers use or might want to rely on models and approaches emphasizing iteration, 
formative evaluation, and deliberation. The field of human resource development concerns a range of non-training 
interventions like coaching programs, on-the-job learning programs, or competency profiles, and thus stresses socio- 
professional design activities that might be reflected in communicative approaches. 

In view of helping students develop the competencies envisaged, it is considered of major importance to 
not only highlight the main domain-specific design models and approaches, but to also address commonalities and 
differences between the various domain-specific approaches, the models, and the four alternative design paradigms. 
Since the beginning of the program in 1981, the faculty has been organized into different departments, reflecting 
different educational domains. Because teachers from all departments participated in the program, input from 
different domains in the curriculum was guaranteed rather naturally. In the current bachelors program, there is more 
explicit attention for the relationship between generic and domain-specific design and these are deliberately aligned. 
This clearly shows in the design ateliers, which are specifically built around a certain theme, and where generic 
design activities are complemented with domain specific design approaches. In the first year of the course 
‘Educational Design: An Introduction’, there is a ‘professor parade’. During these sessions, six professors provide a 
two-hour sketch of their educational domain by illuminating specific design and research interests, themes and 
activities in relation to the generic model, and by explicitly focusing on the merits and shortcomings of the generic 
model for their domain. The masters further concentrates on domain-specific design approaches and principles. 

Design Principle 4: Explicit Coupling Of Educational Design And Research 

A recent development in the academic world is the emergence of and growing attention to ‘design-based research’ 
or ‘design research’ (van den Akker, Gravemeijer, McKenney, & Nieveen, 2006; see also Richey, Klein, & Nelson, 
2004; Burkhardt & Schoenfeld, 2003; Design-Based Research Collaborative, 2003; van den Akker, 1999). Design 
research may be characterized as follows (van den Akker et al., 2006, p.4): 

• Interventionist: the research aims at designing an intervention in the real world. 

• Iterative: the research incorporates a cyclic approach of design, evaluation, and revision. 

• Process-oriented: a black box model of input-output measurement is avoided; the focus is on understanding and 
improving interventions. 

• Utility-oriented: the merit of a design is measured, in part, by its practicality for users in real settings. 
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• Theory-oriented: the design is (at least partly) based upon theoretical propositions; and field testing of the 
design contributes to theory building. 

Following Barab and Squire (2004), van den Akker et al. broadly define design research as “a series of 
approaches, with the intent of producing new theories, artifacts, and practices that account for and potentially impact 
learning and teaching in naturalistic settings” (2006, p.5). They describe three motives for initiating design research: 
the desire to increase the relevance of research for educational policy and practice, the development of empirically 
grounded theories, and the aspiration to increase the robustness of design practice. Along with the pragmatic design 
paradigm, design-based research exemplifies the growing importance of formative evaluation during the design 
process. 

Since its start the program has included courses on research methodology along with design methodology. 
Research methodology courses deal (amongst other things) with the development and use of a consistent ‘research 
chain of reasoning’ (cf. Krathwohl, 1998) which is the core of the so-called empirical cycle. With the growing 
interest in design research, what should become more manifest in the program is that the research chain of reasoning 
will gradually reflect academic skills that are also vital to the educational engineering processes. Design research 
requires the educational designer to conduct a systematic preliminary investigation of tasks, problems, and context. 
This includes searching for more accurate and explicit connections of that analysis with state-of-the-art knowledge 
from literature; formulating a guiding problem statement; reasoning logically towards a solution based on design 
principles derived from the preliminary investigation; and submitting the blueprint or a draft of intervention to a 
formative evaluation that uses the design principles as evaluation criteria. Figure 2 visualizes the analytical 
relatedness between research and design methodology. 

Figure 2: The relatedness between research and design methodology. 




In the bachelors program’s research stream, students are taught basic research methodology with a primary 
focus on the quantitative research paradigm, and are trained in more concrete quantitative techniques for data 
gathering and analysis. This culminates in the third-year research assignment, where students in the role of junior 
researchers are expected to conduct a small, yet well-defined part of research in their chosen design context. 

The design ateliers, the third-year authentic design project, the masters course assignments, and the final masters 
project provide students with the ultimate opportunity to integrate research and design. For example, in Atelier 2, in 
line with pragmatic approaches, students are guided through an analysis phase and through three cycles of design 
and evaluation (screening, expert appraisal, and try-out) while developing a web quest. In Atelier 5, students are 
expected to hand over an evaluated design. In this respect, the approaches in the ateliers more or less have 
characteristics of design research (interventionist, iterative, process-oriented, utility-oriented). 
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The Use Of A Design Model 

In order to meet all the design principles as elaborated before, we view it as important to use a guiding educational 
design model that can serve as a ‘kapstok’. In the starting years of the program, we relied on a model, formulated by 
Plomp (1982). A typical approach of this procedural model is that an educational problem - defined as a 
discrepancy between ‘what is’ and ‘what should be’ - is tackled via a five-stage (ADDIE; actually: ADDEI) general 
procedure for problem-solving by means of a systems approach. 

A validation study of this model by Pieters & Bergman (1992) showed that designers thought that the 
visual representation of this model was too linear and too instrumental in nature, and that it failed to embody other 
approaches. A remolding of this model was found imperative in order to achieve the following (Verhagen 2000): 

• stress the implementation perspective more, exemplified by the motto that implementation starts in the 
preliminary investigation phase and should be continuously reckoned with throughout the design process; 

• accentuate the role and importance of formative and summative evaluation, based on the adage proclaimed 
throughout the program that there cannot be analysis, design, development, and implementation without 
evaluation; 

• represent all four alternative design paradigms and underlying rationalities distinguished by Visscher-Voerman 
and Gustafson (2004); 

• provide room to typify models and approaches that are domain-specific. 

The converted model has been called the Verhagen, Kuiper & Plomp Model and has been pictured in 

Figure 3. 

Figure 3: The Verhagen, Kuiper & Plomp model (Verhagen, 2000). 
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The Quality Of The Bachelors And The Masters Curriculum 
The Envisaged Quality From An Academic Perspective: The External Audit 

The quality of a program - in terms of its efficiency and effectiveness - depends on how it is enacted, how it 
is experienced by students (in terms of learning experiences and outcomes), and how it is perceived by teachers and 
other stakeholders. Indications about some of these variables are available from three sources: (1) an external audit 
of both programs (a process that started in 2005 and was completed by the end of 2006), (2) yearly surveys among 
alumni about how well the alumni are doing in their professional practice and what they consider in retrospect as 
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strengths and weaknesses of the program(s), and (3) regular course evaluations as part of the internal quality 
assurance policy. Here and now we confine ourselves highlighting accreditation findings. 

The external audit took place within the framework of the overall transition of Dutch university education 
into a bachelors-masters system. Because of this, all redesigned degree programs have to be externally audited using 
a standard protocol and then accredited to receive finding from the national government. The protocol covers six 
topics: objectives of the program; content and structure of the program; deployment of staff, facilities, and 
provisions; internal quality assurance; and results in terms of the students’ success rate and the level and quality of 
the qualifications achieved by the graduates. These topics are assessed via 21 facets, for each of which criteria have 
been formulated. The starting point is a self evaluation report, followed by on-site visits by an official Audit 
Committee where documents are studied and staff, students, and alumni are interviewed in group sessions. Only the 
qualifications ‘pass’ or ‘fail’ exist for the six topics. More distinct judgments are given for the 21 facets, however. 
Both the bachelors and the masters program got passes for all topics. The Committee was positive about the profile 
of both programs, which was recognized as clearly design-oriented in the tradition of the international domain of 
instructional design and technology, and well articulated in the curriculum. The level of the end qualifications of 
both programs were rated as good. According to the Committee, graduates describe themselves as problem solvers 
at an academic level. The bachelors program was judged as coherent, with the ateliers as an effective device for 
fostering interaction between theory and application. The masters program received some criticism, however. In the 
Committee’s opinion the various tracks are so different that they found it hard to see them as viable options for one 
masters program. That being the case, the Committee judged that the various tracks offer coherent one-year 
programs in their specific domain that comply with the general design-oriented profile for educating academic 
problem solvers. Theory development and research being seen as instrumental to design processes is characteristic 
of this profile at the masters level. The Committee recognized that a proper balance between design and research is 
strived for. In the bachelors as well as the masters program sufficient attention is paid to academic reflection. It was 
also noticed that there exist many professional connections (via staff and assignments) with the world of business 
and industry and with all kinds of institutions and agencies where alumni find jobs (all yielding a clear picture of the 
labor market), but the goals of both programs have been developed without the systematic involvement of employers 
of graduates. 

The Envisaged Quality From A Professional Perspective: The Alumni Survey 

All students (36) who had graduated from the program between 2003 and 2004 have been sent a survey with 
questions in which alumni are asked to look back at the curriculum from the perspective of their current work 
situation. The goal was to check whether changes in the outline and content of the curriculum were necessary or 
regarded to be useful (Slotman & Meijer, 2005). 

The response to the questionnaire was 64% (23 persons), meaning that the response group is only partially 
representative for the target group. 

The questionnaire contains questions relating to the following themes: 
general characteristics of alumni 
education 

current work situation 
current job 

review of the curriculum 
alumni association 

The results show that 95% of the alumni have found a job two years after finishing their studies, 4 of them as a 
PhD-student. More than 80 % of the alumni have found a job 6 months after graduation. The level and nature of the 
job and task description matches the educational domain for which the alumni were educated. 

Respondents are satisfied about the preparation of their studies to their current job. They indicated to choose the 
same studies again as preparation for their current job. 

Discussion 

Above, we have shared the ins and outs of our educational design program, in which we aim at educating 
academically qualified problem solvers who are competent to work in a variety of professional settings. In closing, 
we reflect on five issues that have gotten attention and that will continue to deserve particular attention: the level of 
academic competence achieved by graduates, the orientation on professional practice, the role of reflection across 
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the program, the attention to be devoted to the instrumental design approach versus other approaches, and the 
relationship between design and research. 

Level Of Academic Competence Achieved By Graduates 

Although the Audit Committee came to the reassuring conclusion that graduates qualify themselves as 
academic problem solvers and that masters theses are of good quality, the one-year-length of the masters program is 
a source of concern. Students who enroll the program have different backgrounds, they can be EDMM graduates, 
graduates from bachelors programs other than EDMM such as professional bachelors who graduated from a 
Netherlands teacher education institute (with or without working experience) and academic bachelors from abroad 
(Africa, for instance). It is our experience that many of the enrolling professional bachelors have difficulty in 
attaining the academic level required within the space of only one year. They simply need more time to master 
design and research competencies with sufficient academic depth even if they invested in a pre-masters program to 
qualify for the masters year. This problem of dealing with students with diverse academic skills and attitudes within 
a short time frame has also been recognized by the Audit Committee. According to the Committee, the best (though 
probably politically unfeasible) solution to the problem would be to prolong the masters program by one year. 
Another possibility, in addition to maintaining a strict admission policy, could be to stimulate students to primarily 
focus on competencies they need to work on in view of the end qualifications to be attained. Such a competence- 
based approach begins with students’ talents, accepting that someone cannot be good at everything. It also entails 
flexibility in contents, pedagogical approach, and assessment modes. Such a competence-based approach, however, 
requires a change in teaching practices and, much harder to realize, teachers’ beliefs. 

Orientation On Professional Practice 

The clear focus in the program on professional practices has been exemplified in several authentic learning 
assignments to be conducted for and in interaction with professional clients. Students highly appreciate and value 
this (according to regular course evaluations) and the many professional connections they made did not go unnoticed 
by the Audit Committee. During the assignments, the role models are provided by staff, which consists of primarily 
academic thinkers and designers. However, what is missing in the program is not only a more systematic 
involvement of future employers in further developing program goals (see external audit), but also a more prominent 
role of professional designers in design courses (in addition to the already existing involvement of professional 
clients and the already existing focus on professional settings). The latter could be realized, for example, by inviting 
alumni to be guest lecturers, act as role models in design situations, or to provide insight into possible design 
shortcuts. Thus, they could be an encouragement to students during their development from novice to experienced 
designer. 

Role Of Reflection 

A large aspect of the academic attitude is revealed in its considerable emphasis on reflection. Compared to 
the instructional design competencies for professional designers as formulated by Richey, Fields and Foxon (2001), 
the component of reflection seems to be more prominent in our curriculum. We regard reflection as an opportunity 
to not only optimally shape a situation and pre -consider design solutions (Richey, Fields & Foxon, 2001), but to also 
make designers become aware of their own individual strengths, shortcomings, interests, and basic assumptions. The 
ability to reflect grants a person lifelong learning and is one of the Dublin descriptors (Joint Quality Initiative, 

2004). However, reflection is still too underexposed, especially in the bachelors program even though its importance 
is acknowledged and its merits are recognized. Students are required to prepare two so called ‘reflection papers’ (as 
part of both the first-year introduction course on educational design methodology and the second-year Atelier 3 
course) and to elaborate a reflection section on part of the design report on the third-year Atelier 5 design 
assignment, but the primary focus is on reflection-on-action rather than reflection-in-action (Schon, 1983). In 
addition, the reflections made are more self-evaluative than reflective. This is possibly due to the fact that it is only 
in the third year that students are taught and coached on how to reflect. As a solution to this shortcoming, the 
program has recently decided to pay more attention to the acquisition of reflection-on-action techniques in the first 
year of the program (from 2007-2008 onwards), and to stimulate the use of reflection-in-action techniques while 
working on design assignments in the five ateliers. 
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Instrumental Approach Versus Other Educational Design Approaches 

As stated above, we have chosen to use the instrumental approach as the leading pedagogical device. It is 
our experience that students who already have some design experience are better able to value the various design 
approaches (instrumental, communicative, pragmatic, and artistic). It is interesting to explore why this could be the 
case, and how we could incorporate that into our curriculum. One possibility is that these students already have a 
broader repertoire or a higher level of expertise (as suggested by Jones & Richey, 2000), which enables them to 
understand and value the different models more quickly and thoroughly. It might also mean that the way we teach 
the communicative, pragmatic, or artistic approach should be changed or intensified. After all, it is still possible that 
our current teaching does not connect to the level of expertise of the novice designer (Dorst & Reymen, 2004). In 
the current program, the four design paradigms and accompanying approaches are introduced through readings and 
lectures at the very start of the program, and students are supposed to refer to these approaches in their reflections 
throughout the program. However, it could be more effective to request students to deliberately think or work as a 
designer from a specific approach (in authentic assignments or in studio assignments, for example) and then ask 
them to compare their experiences with this approach to experiences with the instrumental approach. Another option 
could be to introduce design approaches other than the instrumental approach later in the program when students 
have really mastered the instrumental approach. From a pedagogical point of view, it might then be easier for them 
to value and interpret the different approaches. lets meer beschrijven als een interessant onderwerp voor toekomstig 
onderzoek. 

Relationship Between Design And Research 

The research courses delivered in the program expose the students to traditional research methods with a 
relatively large time investment in elements of quantitative research such as quantitative data gathering methods 
(surveys) and analysis (SPSS). This is motivated by the fact that students consider these subjects rather difficult to 
master. At the same time, we experience that students have difficulties in recognizing the relevance of the research 
approaches that are taught to them for design activities. One reason is that research methodology courses and design 
methodology courses are still more or less separated streams in the curriculum. There is especially room for 
improvement at the bachelors level as far as the link between research and design methodology is concerned. 
Improvements to be seriously considered are twofold. First, for students it still should be made more manifest that 
the research chain of reasoning also reflects academic skills that are vital to educational engineering processes. This 
idea has been formulated as a redesign principle in the above, but still deserves further attention. Second, more 
attention is needed in both programs for the concept and methodology of research-based educational engineering 
(van den Akker & Kuiper, 2007). 

In this article, we have shared the ins and outs of our educational design program, in which we aim at 
preparing students to work as academic or practitioner in the field of educational design and technology. The Audit 
Committee has provided evidence that the current bachelors and masters programs are of high quality. They are 
well-balanced, and consistent with the redesign principles. Through the programs, students do indeed develop into 
academically qualified problem solvers who are competent to work in a variety of professional settings. The critical 
remarks made in the above final paragraphs are just meant to illustrate that we take our job as designers and teachers 
of the program very seriously, and are constantly striving for the further improvement of program quality. 
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Teaching Systematic Reflection To Novice Educational Designers 

Irene Visscher-Voennan & Henk Procee 

Summary 

How to help students in the field of instructional design and educational technology to develop their professional 
expertise through systematic reflection? This question is answered by describing the intended, implemented and 
attained curriculum of a third year university bachelor course on systematic reflection for design students. In this 
course, students learn four modes of reflection that originate in the work of the philosopher Kant. This approach is 
perceived as a different, yet productive addition to existing reflection approaches. 

Introduction 

Reflection is an important competency for designers in general (e.g. Schon, 1983) and for instructional or 
educational designers in particular (e.g. Rowland, 1993). Moreover, in the process towards formulating instructional 
design competencies, expert designers viewed reflection as an “essential element of successful design for all 
designers, novice and expert” (Richey, Fields, & Foxon, 2000, p.72). Since it is such an important competency, 
reflection should be a clear component in the education of designers (e.g. Rowland, Fixl & Young, 1992; 

Shambaugh & Magliaro, 2001). 

Reflection can be regarded as an element of academic competence. Therefore, in higher and university 
education, teachers regularly ask students to reflect on their work (e.g. Boud & Walker, 1998; Boud, Keogh & 
Walker, 1985; Korthagen & Vasalos, 2002). In a number of cases, teachers provide guiding questions to steer the 
students’ reflection. These guiding questions frequently are based on a logic of improvement: What was wrong in 
your project, what are the causes of it, how can you learn to do it better a next time? In line with these questions, 
however, more than occasionally the student papers turn out to be rather an expression of a negative self-evaluation, 
than of real reflection. Not only can this approach be demotivating since students need to start with the idea ‘I did 
not do my work well’, but it also leads to superficial professional and academic growth, resulting in a mastery of 
explaining their failures in terms of external circumstances, and at best to only technical clues for how to improve 
their work. Some teachers might approve this as good reflective work, others will not. It thus shows that the concept 
of reflection is vague, meaning different things for different persons, and that students have difficulty in doing it. 

This observation has resulted in a collaborative endeavor of a philosopher and an ID expert , being the 
authors of this paper, to develop a distinct course on systematic reflection for students in the field of instructional 
design and educational technology at the University of Twente. In this course, students, who are in the third year of 
their study, learn four modes of reflection that originate in the work of the philosopher Kant. The course has been 
taught to 7 groups of students since 2002-2003. The student study load for this course is 5 European Credits, 
equaling 140 hours. 

The basic question underlying course design and delivery has been: How to help students in the field of instructional 
design and educational technology’ to develop their professional expertise through systematic reflection? 

The purpose of this paper is to provide an answer to this question by describing the intended, implemented 
and attained curriculum (Goodlad, 1984; van den Akker, 2003). The first section of this paper is therefore devoted to 
our views on the nature of reflection and how to teach it (the intended curriculum), a description of the course as it is 
implemented (the implemented curriculum), and an overview of student perceptions and results (attained 
curriculum). 

From the beginning we took a design research perspective (Nieveen, McKenney & van den Akker, 2006) in the 
sense that we: 

• expressed and described a conceptual framework, based on literature review that portrays our perspectives on 
reflection and on teaching reflection; 

• conducted an iterative course planning, in the sense that evaluation results of a course in one period led to 
revisions of the course in the next period; 

• systematically documented and reflected upon the process and its outcomes, in order to support retrospective 
analysis. As such, for each course we rely on the following documents: course syllabus, author(s)’ articles, 
electronic learning system, teacher planning sheets and log files, e-mails between teachers and between teachers 
and students, course evaluations -both smile sheets and in-depth evaluations- by students, student papers, 
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written feedback on student papers by teachers and students, student grades, and external audit statements on the 
quality of the course. 

In the second part of this paper, we will describe these sources and explain how the design research perspective 
supports our work. An explanation of the methodology and evolvement of the course are followed by a discussion of 
the findings and future implications of our work. 

Part 1: The Curricular Representations Of The Systematic Reflection Course 

The systematic reflection course will be described at different curricular levels. First, the basic assumptions 
and views underlying the course will be formulated as the intended curriculum. Second, the section on the 
implemented curriculum will describe how the course is being organized and implemented. Third, in the section on 
the attained curriculum, we describe students perceptions of the course, and describe their development in the 
reflective competence. 



The Intended Curriculum 



The Nature Of Reflection 

According to John Dewey (1916, 1933), by many authors perceived as the founding father of reflection in 
education, reflection starts with experience, not with theory. “An ounce of experience is better than a ton of theory, 
simply because it is only in experience that any theory has viral and verifiable significance” (Dewey, 1916, p. 44). 
Dewey distinguishes between two types of experience. The first is trial and error, leading to rules of thumb without 
insights into acting and outcomes. The other is reflective experience, meant to get insights into relations between 
causes and effects both in theory and in acting. Dewey’s model for reflective experience can be viewed as a circle, 
consisting of the steps: sense of a problem, the observation of conditions, the formation and rational elaboration of a 
suggested hypothesis, and the active experimental testing. This approach to reflection can be characterized as a 
process in which the student acts as a problem solver (or in a metaphor: as an engineer) and grows through learning 
from his or her own mistakes. 

One key characteristic of Dewey’s model, and a lot of models stemming from this tradition, is that it is 
aimed at improvement. Although the wish to improve one’s performance is instrumental to professional growth, this 
approach has, as already said, a serious drawback: the learner must take a negative view towards his or her previous 
experiences. As such, reflection has the nature of a (negative) self evaluation (‘I did not do well’), often resulting in 
excuses and good intentions (‘I will do better next time’). Also, students often stick to giving explanations of why 
something did not go well. This way, they do hardly get any new and deeper insights. 

As described in previous publications (Procee, 2006a; Procee, 2006b) an in-depth study of the work of the 
philosopher Kant (1787, 1956) results in another, more fruitful view on reflection. Rather than a focus on 
improvement, it emphasizes the making of discoveries. Kant distinguishes between understanding (‘Verstand’), 
judgment (‘Urteilskraft’), and experience. On the one hand, there is the experience, something the individual has 
done or encountered. On the other hand, there is understanding, which is related to the ability to grasp logical, 
theoretical, and conceptual rules; in-between, there is judgment, which is related to the ability to connect 
experiences with rules. Folllowing Kant, there can be posed two assertions: 

• Learning formal knowledge should be characterized in terms of understanding. 

• Exercising reflection has to be characterized primarily in terms of Kant’s notion of judgment. 

Following these assertions, the character of reflection radically differs from the character of learning formal 
knowledge. Because judgment (as the capacity to combine heterogeneous elements) is situated between experiences 
and understanding (concepts), reflection activities depend on the breadth of the experiences, as well as on the 
feasibility and productivity of the concepts introduced for inquiry into those experiences. This mirrors Kant’s 
famous dictum: ‘Concepts without experience are empty, experiences without concepts are blind’. In the process of 
reflection, concepts have a double function; they function as a source of inspiration to analyze the experience and as 
an outcome in which they are better understood than before (Procee, 2006a). According to this approach reflection is 
not submitted to the logic of improvement, but to an emotionally more neutral logic: the logic of making 
discoveries. 

As argued by Procee (2002) Kant’s table of moments of thought, as developed in his Critique of Pure 
Reason, can be usefully adapted and developed for a systematic approach to reflection. Kant’s four moments are: 
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• Quantity: this moment creates a reflective space that stimulates learning discoveries. It generates new and 
unexpected views on experience. 

• Quality: refers to points of view that may be helpful to estimate (elements of) experiences and choices made. 

• Relation: this moment brings about dynamic elements by introducing points of view that are related to different 
visions from a professional as well as a social context. 

• Modality: this refers to the status of the judgment, in our view, it reflects on the reflection process itself and on 
aspects of (professional) identity. 

In order to make these modes of reflection communicable, we introduced geometrical names; point reflection 
(quantity); line reflection (quality), triangle reflection (relation); circle reflection (modality) (see Procee & Visscher- 
Voerman, 2004; see table 1). 

In the following sections, the general structure and aspects of the four reflection modes are described. Step- 
to-step approaches of each mode will be described in a later section, where the content of the course events are being 
described. 



Table 1: Reflection modes after Kants moments of thought 



Kant’s moments of thought 


Type of reflection 


Geometrical figure 


Quantity 


Point 


• 


Quality 


Line 




Relation 


Triangle 


A 


Modality 


Circle 


O 



Point Reflection 



Underlying this mode of reflection is the acceptance that an experience should not be seen as an 
unequivocal fact, but that it may be described in many different ways. By the ‘free play of imagination’ it is possible 
to highlight unique features. Therefore, it is important to take the experiences to a new reflexive space outside the 
own experience, in which new thinking activities can take place. In the reflexive space all kinds of discoveries can 
be made, for example with respect to the use of materials (‘what is the function of different materials’), personal 
feelings (what in the project made me happy, what made other people happy), or a color (green: ‘where did I appear 
to be a greenhorn?’ ’What was fresh and new in our approach?’). The point reflection is visualized in figure 1. The 
choice for the external reference point depends on the pragmatic question: do I expect to learn from this? But one 
should be careful: each question will highlight different aspects in the experience, and some questions are more 
meaningful than others. Choosing a topic close to the experience will not add to deep reflection. For example the 
question ‘how was the communication with the client?’ will lead to a chronological description of what has been 
done, with little room for new viewing points. On the other hand, viewing points far from the experience may raise 
so many new questions, that it results in chaos. For example, the topic ‘design vision’ may lead to reflections on this 
vision itself, on parts of the vision, on other design visions, on how the vision permeates the product, etc. There are 
than too many questions, so that the link to the direct experience disappears. 

Apart from the viewing point chosen, the quality of the reflection also depends on the type of questions that 
are being raised. ‘Why’ questions can better be avoided, since they provoke a justification for actions, as if the 
experience itself was wrong, which may strengthen the negative feelings of the student about the performing of his 
or her task. Questions such as ‘how’ and ‘what’ are better, because they put less pressure on the reflecting person. 




In this stage the role of the reflection partner is precarious. Such a partner has to be involved in the learning 
process of the student, and less in the aims of the design project itself. Reflecting with colleagues who have been 
working on the same project has some disadvantages, because they share the similar experience. Reflecting with one 
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self, although being a real human capacity, is even more difficult. To create a model of the reflection partner (and 
also of the teachers involved in these reflection processes) a novel of the Swiss author Max Frisch is very helpful. In 
the novel Mein Name sei Gantenbein, Frisch (1964) describes a man who is thinking about living as a blind person. 
Fie is curious to know what could be his most adequate vocation. In a great passage he concludes that the role of tour 
guide would be most appropriate. Suppose, you are seated at the foot of the Acropolis and your group of tourists 
have to explain you what they have witnessed. In that case they will observe many times better than in situations in 
which the tour guide is a seeing person. This image - the blind tour guide - is the bench-mark of the reflection 
partner. It is a person who does not tell his or her own interpretations and narratives, but who inspires other persons 
to tell their experiences in depth. 

Line Reflection 

Line reflection is about quality. It borrows its name, and partly its structure, from the philosopher Plato. 
Plato discriminates between the factual reality and our usual experiences on the one hand, and eternal criteria in the 
world of Truth, Goodness, and Beauty on the other hand. 

Thus, people have different views of quality; they can qualify a situation or action as ‘good’ or ‘bad’, and 
are able to indicate why they do or do not appreciate it. In their minds, they have a view of the ideal or perfect 
situation or action, against which they mirror the situation or reflection at hand. These views may change over years, 
based on new experiences and new insights. Also, one vision or norm could be more fruitful than others. In this type 
of reflection, experiences are judged against own norms and criteria (see figure 2). But, proposed criteria, norms, 
and ideals also have to be examined on their appropriateness. For that reason, this kind of reflection differs in some 
important aspects from evaluation. It does not accept criteria as given from the outside. It starts, instead, with a 
procedure in which one defines and scrutinizes his personal criteria in view of his learning process. Also, based on 
the reflection, personal norms and criteria may be adapted. 

Figure 2: Visualization of the line reflection 
The ideal situation or action 



I t 

The factual situation or action 



Triangle Reflection 

Triangle reflection is the most dynamic form of reflection. It is based on the theory of semiotics put 
forward by Charles Morris (1938). In the triangle reflection, relations and connections are put central. People 
interpret facts or events in a certain way (for example against the norms as described in the line reflection), 
depending on their experiences and knowledge. For example, in traffic a red light is a sign to stop, a green light is a 
sign to drive. In such daily structures, there are three elements connected: the experience or fact (sign or 
designatum), the person(s) who give(s) meaning to the sign (interpreter), and the frame of interpretation 
(classifications, theories) from which meaning is derived. This can be visualized in a triangle (see figure 3). 

Different interpreters may interpret the designatum in different ways, depending on their own frame of 
reference, and based on that, they may come to different actions. An interpreter can be an individual, but also a 
social group (cultural group, professional group, religious groups, scientific groups, etc). Such groups have their 
own frame of reference according to which they describe and explain reality. 

Characteristic for this model is its dynamic structure. A change at one angle does have repercussions for the 
other angles. A change in the frame of interpretation (for example: the alteration from ‘smoking is fine’ to ‘smoking 
is bad’) does change situations in reality (special smoking places) and also creates a different image of persons (a 
smoker nowadays is an irresponsible creature). In a similar way, a change in reality, especially technical artifacts 
such as mobile telephones, redefines the social community (users and non-users), as well as the frame of 
interpretation about ‘normal’ communication. 
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Figure 3: Visualization of the triangle reflection 
Frame of interpretation 
(classifications, theories) 




interpreter designatum 

(individual/group) (elements in reality, experience, facts) 

Circle Reflection 

According to Kant the moment of ‘modality’ has a weaker position than the other moments, because it is 
about the way of thinking itself. In his words: ‘The modality of judgments is a quite peculiar function. Its 
distinguishing characteristic is that it contributes nothing to the content of the judgment (for, besides quantity, 
quality, and relation, there is nothing that constitutes the content of a judgment).’ (Kant, CPR, B99, 100). For that 
reason, we interpret this moment as the cyclical character of reflection itself. Reflection starts with experience, 
detaches itself from it, examines a variety of quality aspects in it, scrutinizes different types of relations, and 
eventuates in a growing capacity to handle new experiences. This is a never-ending cyclical process which can be 
put forward on the three other reflection techniques. Circle reflection can be performed on different levels: 

• the design project or the design experience (what reflection method did 1 use to study the design experience? 
What new and unanswered questions with respect to design issues do I need to investigate further?) 

• the professional identity (how does my professional identity fit to my personality? What is in my professional 
toolbox and what is lacking? What types of design problems do fit my professional identity?) 

• the reflection itself (what types of reflection did help me to gain new insights? What reflection methods do 1 
need to study more?) 

How To Teach Reflection 

Also from Kant, we derived three basic assumptions about the pedagogical role of the teacher, which are 
central in our approach. Firstly, in judgment-oriented educational settings, such as in case of reflection, the 
individual student and his/her learning goals is central. It asks that students take on an autonomous attitude and 
teachers a coaching attitude. In an environment where students are used to teachers taking the lead, this means a 
radical change. It asks for an explicitly active rather than passive role of the student. It is the task of the coaching 
teacher to improve the emotional trust and self-confidence of students to take the lead. 

Secondly, judgment is related to the individual. Thus, reflection is an individual activity, in which the one 
who reflects takes the lead. Teachers facilitate reflection, but should not judge the experience of the learner, since 
that provides the materials to reflect upon. The student judges his or her own learning experiences in reflection 
activities, but they also judge their personal and professional growth through the reflective activities themselves. In 
the course, this shows in students making a justified suggestion for their own grades. 

Thirdly, teachers do not instinct specific content, but put instead the reflection competence central. 
Students choose themes or topics from their design experience that may result in further personal and professional 
development. Teachers provide new viewing points or help students find these, to steer their reflective actions. 

The Implemented Curriculum 

In the course Systematic Reflection, we have incorporated the above mentioned principles. Students learn 
how to reflect, deeper than just common sense thinking, aimed at their own professional development. In this 
process, we follow a Kantian approach. As teachers we act as coaches. We facilitate the reflection process, while 
leaving the judgment of the experience to the students. As content for the course, students take their own design 
experience from a design project that they recently finished. In this project, atelier 5 (see also Visscher-Voerman, 
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Kuiper & Verhagen, paper, this AECT), students work in groups of three or four students for an external client, e.g. 
a school principal or school teacher, a museum, a manager in a company, a teacher trainer, etc. 

Below, we will describe the course, addressing the elements from the curricular spiderweb (van den Akker, 
2003), and provide step-by-step approaches for each of the reflection modes in the subsection content. 

Goals 

The following goals have been formulated for the course. The student demonstrates that: 

• He/she has made an intellectual growth in reflection throughout the course; 

• He/she is able to apply the modes of reflection correctly; 

• He/she is able to apply reflection tools, partly with using literature, with the function of gaining new insights 
regarding his/her own functioning as a professional 

• He/she is able to support peers in their reflection processes by providing concrete and supporting feedback. 
Course Outline 

Course moments have been centered around the four reflection modes. Each reflection mode follows the same 
structure: 

• Two-hour tutorial in which the reflection mode is introduced and a systematic step-by-step approach is being 
provided; instruction of the steps is alternated with small exercises, or teacher role play. 

• Students write a reflection paper according to this specific mode, starting from an individual relevant viewing- 
point, and relating it to their own specific design experience. 

• Both teachers and two peer students provide individual written feedback, sent to the students by mail, in 
advance to the feedback session. 

• Two-hour feedback session, in which teachers provide overall feedback on the reflections, for example, by 
addressing topics that students chose to reflect on; by identifying pitfalls for reflection in the specific mode; by 
making suggestions for how to make the reflections in the specific mode more powerful, etc. 

• Students use the peer and teacher feedback to improve their papers into the final paper which is being assessed 
and discussed in an individual oral conversation with the teachers. This oral conversation replaces the last 
feedback session related to the circle reflection. 

• One element in this conversation is a student’s proposal for grading his or her own work. 

Course Content 

The Tutorial On Point Reflection 
This meeting addresses the questions: 

• What is reflection, and why is it important? 

• What are characteristics of good questions? 

• What is the structure of the point reflection? 

• How do you choose external points of view to start the reflection from? 

• How can you help others to reflect as a reflection partner? 

The step-by-step approach of the point reflection is: 

1 . Reflection can be initiated by using many different viewing points, for example, theoretical concepts, 
contradictions, proverbs and sayings, colors, emotions, tools, sports/games, ‘faith, hope, and love’, title of a 
roman, etc. Picking such a viewing point rather than project related words (such a goal, communication, 
instructional strategies) helps to come loose from one’s usual patterns. 

2. Open up your creativity by generating questions related to different viewing-points chosen from the list above. 

3. Choose a viewing point, that highlights your experiences in new ways, and that seems a powerful vehicle for 
learning; 

4. Reflect, not on a viewing point in the experience, but from a viewing point. Avoid raising ‘why-questions, but 
concentrate on ‘how’ and ‘what’ questions. 

5. Summarize what you learned. 

In the tutorial teachers make an inventory of student former reflection experiences; students conduct a 
reflective conversation in pairs; we read aloud a Fairy tale Spirit in the bottle to illustrate what reflection is 
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( http://www.pitt.edu/~dash/grimm099.html ); We practice the first two steps of the step-by-step approach. Also, the 
theory of reflection, according to Kant is instructed. As home work students are asked to generate at least 5 
questions from 3 viewing points and typify project situations. 

The Tutorial On Line Reflection 

The guiding questions for the tutorial on line reflection are: 

• What is the structure of line reflection? 

• What are the similarities and differences between line reflection and evaluation? 

• After the tutorial, students are expected to be able to write a line reflection. 

The step-by-step approach for the line reflection has three different phases. In the first phase, as 
preparation, norms and quality criteria are being formulated. The second phase consists of the reflection itself. In the 
third phase, the student discussed the consequences, either being related to planned adjustment of acting and 
performance, or adjustments of the norms themselves. 

1. Choose a professional role (for example, teacher, designer, researcher, or advisor). 

2. Distinguish between actor, process and product (for example: designer, design process and designed product) 

3. Formulate for each of them norms or quality criteria, starting from questions such as: ‘What are characteristics 
of a good designer?’ ‘What are criteria for a good design process’? ‘When are you satisfied about the product?’. 
List these norms in a scheme (see an example below). 

4. Choose one of the norms, guided by your intention to get important learning experiences. 

5. Develop this norm further into norms of quality, first based on your own ideas, expectations, and experiences; 
and then by making use of literature. 

6. Relate these norms to your own experience. 

7. Summarize what you learned, both with respect to schemes for changing performance and acting and to 
adjustments of norms. 



A good designer . . . 


A good design process . . . 


A good product . . . 


Is a good listener 
Is creative 

Applies design models 
Knows the context 
Can motivate people 
Is a good project manager 
Is analytical 


Is based on scientific knowledge 
Relies on users 

Makes use of formative evaluation 
Lias an implementation perspective 


Meets the specifications 

Is accepted by the client 

Is accepted by users 

Yields learning effects 

Flelps to solve the original problem 



During a first brainstorm there will not be a direct horizontal link between the columns. This might be 
reached in a further refinement of the scheme. 



The process of line reflection resembles the process of evaluation, but is basically different on several 
aspects. Firstly, because the student himself determines what norms to study, secondly because its goal is to learn 
and not to judge. Thirdly, the norms are not viewed as a given, but are investigated on their quality. Also, not only 
the quality of the final product is put central, as is usually the case in evaluation, but also that of the design process, 
and especially the actor himself. 

After a short introduction to the line reflection, we practice with the group steps 1 to 4, by focusing on the 
role of researcher; In pairs, students practice step 5; Students exercise difference between line reflection and 
evaluation, by generating 5 questions stemming from that norm that have a reflective nature, and 5 questions with an 
evaluative nature. As home work, students are asked to write a line reflection paper, starting from a norm that is 
interesting from their individual point of view, from 6-8 pages. They are expected to use literature to specify their 
norms and/or to support their summary of what they have learned. 

The Tutorial On Triangle Reflection 

The goal of this session is to make students familiar with two types of triangle reflection, related to 
stakeholder analysis, and to scientific approaches. The step-by-step approach related to stakeholder analysis is: 

1 . Choose a theme, fact, or situation for the reflection (designatum) 
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2. List all (groups of) ‘interpreters’ related to your design project in general; 

3. Delete all interpreters from your list, that are not worthy for your chosen theme. You end up with a list of 
stakeholders. 

4. Formulate hypotheses about the frames of reference for each (group of) inteipreter(s); 

5. Test your hypotheses (not always possible) 

6. Interpret design experiences in relation to these differences 

7. Analyze how to deal with differences in frames of reference (what strategies are possible, do fit your 
experiences and person, what tools do you need for that) 

8. Summarize what you have learned. 

The step-by-step approach related to scientific approaches is: 

1 . Choose a theme, fact, or situation for the reflection (designation) 

2. choose one or more different theories about that theme, for example, design theories, instructional theories, 
communication theories, etc) 

3. Formulate hypotheses: describe from the point of view of each theory how the chosen theme was present in 
your project, or should have been present; 

4. Interpret your design experiences in relation to these hypotheses and theories; 

5. Analyze how to deal with the differences between the theoretical frames of reference (what strategies are 
possible, which ones do fit your experiences and person, what tools do you need for that); 

6. Summarize what you have learned. 

In the tutorial, first both step-by-step approach is explained. Then, we practice the stakeholder approach 
steps 2-4 related to the theme of quality of the product. Then, with the whole group, we practice the approach related 
to scientific approaches, starting from instrumental, communicative, pragmatic design approach (Visscher-Voerman 
& Gustafson, 2004). As homework, students are asked to write a triangle reflection for one of both options, 6-8 
pages, use of theory. 

The Tutorial On Circle Reflection 

As a preparation for the course, each student reviews his work as done in the course thus far and makes a 
(small) list of topic(s) he/she wants to study during the circle reflection. For the circle reflection we do not present a 
clear step-by-step approach. Instead, we discuss the table as presented in table 3. The scheme is an instrument to 
visualize how one can reflect on different levels, ranging from the evaluative level to a more philosophical level 
(depth of reflection); it also shows how reflection topics can be related to the profession, the individual professional, 
or the wider environment (breadth of reflection). During the tutorial we first formulate questions related to the theme 
of communication. Then, we help students to formulate questions related to their theme of the circle reflection on a 
higher level. 



Table 3: Breadth and with of reflection 



Breadth of reflection 


Profession or 


Self 


Wider environment 


Depth of reflection 


problem 






3. epistemic/critical 
{philosophical/ contemplative) 








2. interpretative 








1 technical/pragmatic level 
( problem solving) 








0. evaluative level 
{measuring effects) 









Feedback Sessions 

In the first session, related to the point reflection, a communication specialist comes in to instinct students 
on how to provide feedback. The other feedback sessions are used by the teachers to provide general feedback 
related to the individual reflection papers. Individual feedback has already been e-mailed to the students by then. 
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Feedback relates to readdressing the specific function and steps of the reflection mode; the role of theory, choosing 
the theme, the level of reflection. 

Course Materials 

Specifically for the course, a short syllabus has been written, that provides background information about 
reflection as summarized at the start of this paper, and that mainly describes and elaborates the different reflection 
modes, with their underlying philosophies. Also a reading guide for the book Procee (1997) is provided. As options 
for students to steer their reflections, some basic literature is suggested, that students could choose to use for their 
reflections (e.g. Curren, 2007; Procee, 1997; Schon, 1983; Simon, 1967; Wouters, 1999). But, what students 
ultimately choose to use, depends on their experience and chosen focus for the reflection. Therefore, in addition, 
students search for scientific or professional literature that connects to their reflection questions. 

Teacher Roles 

In the first part of the paper we have already described our teaching philosophy. In addition it could be 
added that throughout the years, we have developed clear and distinct roles. The role of the philosopher is to instinct 
the reflective theory, to provide new viewing points, to inspire students. The role of the ID teacher is to keep track of 
course planning and to manage the time, to provide design knowledge, and to communicate with students. Both 
teachers alternate the introduction, leading, and discussion of the hands-on experiences during the tutorials, and both 
teachers provide feedback on papers. 

Assessment 

In an oral meeting with each individual student, we discuss his or her work. As input for this meeting is the 
student’s paper, consisting of four different reflection papers that has evolved through the course and has been 
improved based on feedback from peers and teachers. An important input for the assessment is the student’s own 
argumentation for his grading. In the student’s argumentation towards the grading we expect -again- a reflective 
approach. The student’s subjective judgment indicating the own learning curve, is compared to the teachers’ more 
external and comparative grading. In most of the cases, both gradings match, indicating that students are very well 
able to judge their own qualities as a professional. Because of that aspect we have chosen for this kind of 
assessment. In more formal education situations we use a different approach, according to the standards of the 
academic forum. 

Changes In The Course Over Years 

This general outline of the course has been rather stable over years, although there were some minor 
changes, based on experiences and student evaluations. Most changes were implemented in 2004-2005 as a reaction 
to two years of evaluation and experiences. Some of these changes were undone or adapted later. Major changes 
relate to: 

• Final oral meeting: in the course 2004-2005 we decided to skip the oral final meeting, in order to save teacher 
time. Although the general appreciation of the course did not change and student results were not different from 
other years, several students indicated that they really missed this meeting, based on what they had heard of it 
from previous years. Therefore, we decided to bring this element back, to bring the course for each individual 
student to an individual end. 

• Peer feedback: In 2003-2004 we were dealing with a very large group (56). Since we had experienced that 
providing feedback on all papers ware very intensive, we looked at possibilities to bring in peer feedback. We 
soon learned that this can be very motivating, and that it can have a positive effect on student attitude and 
achievement (Topping, 1998; see also Van den Berg, Pilot & Admiraal, 2005), although we have not 
deliberately measured this effect ourselves). The peer feedback has become an essential part of the course, and 
is included in the formulation of the course goals. Different from the first time, in the course 2006-2007, 
students were now explicitly instructed on providing good feedback by a communication teacher. 

• The amount of teacher feedback: in 2005-2006, a year after peer feedback had been introduced, the teachers 
only provided general feedback in the feedback tutorial an no individual feedback. This was evaluated as a weak 
point during that course. Students indicated to miss this external reference, and also indicated that not every 
student was as good in giving feedback as others. The following year, teacher feedback was included again. 
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• Point reflection: the first two years, the concept of point reflection remained rather vague, as also showed in the 
student evaluations. Students found it difficult to step outside their own project and the teachers had no good 
clues of how to inspire them. We mainly used this meeting to ask students to come up with a reflection agenda. 
Two years ago, we experimented with the use of sayings, proverbs, sports, colors, emotions, etc. to ask students 
to generate inspiring and unique questions. The use of these unusual viewing points opens up students’ 
creativity and helps them to step outside the experience. 

• Order of the reflection modes: the order of line reflection and triangle reflection is rather arbitrary. Some 
students have a natural inclination to the first mode (especially the more evaluative types among them), others 
(who are inclined to discovering differences) prefer the last mode. During some years we started with line 
reflection, in other years we started with triangle reflection. The order made no differences in the end. 

• Place of reflection in the curriculum: In their evaluations students indicated several times that they would have 
wanted to be familiar with the reflection techniques in order to steer their design activities. Therefore, in 2004- 
2005 we decided to offer the triangle reflection mode at the start of a design course, as part of stakeholder 
analysis. At the end of the course, it turned out that none of the design groups had made explicit use of this 
technique during their project. Students indicated they were not yet ready for those techniques, since the project 
had so many new aspects they needed every attention to control the design process (design in real context, for 
real client, real communication). Stepping outside the process to reflect was not yet an option for them. 

• The roster: in the first two years, the tutorials and feedback sessions were planned at the same day (4 hours). 
Students indicated that that session was too long to stay motivated and attentive. They also indicated that the 
time to write a paper was very short. Therefore, in 2004-2005, we split up both sessions. Students could hand in 
their paper during the next tutorial. Drawback: they could already start writing before they had their general 
feedback. 

• Step by step approaches: In the beginning years, we divided each reflection mode in three steps (preparation, 
reflection, and summary of learning). Students indicated to find it still difficult to reflect and that reflection 
remained rather vague, and that they needed more guidance and structure. Therefore, we have extended the step- 
by-step approaches for each of the reflection modes, by distinguishing more clearly the different steps. Although 
we were rather hesitating to do so, since they might see it as a simple checklist and thus might hinder them to 
think for themselves, students appeared to get support from this, and do not report about vagueness any more. 

The Attained Curriculum 



Student Perception Of The Course 

Students are very positive about the course. On a scale from 0 to 10 students the mean student approval each year 
has been 7.9 or 8.0. No student ever gave an unsatisfactory mark (e.g. 5 or lower). 

Students especially highly value the content of the course and the pedagogical approach underlying it. 
Students identify this course as extremely supportive for their own professional development. They like the different 
teaching pedagogy, although this, at the same time is difficult for several students. 

Student Growth In Reflection 

All students increase their reflective competence during the course, where some make impressive progress. Overall, 
over the years, student results have not improved, it even seems that grades are a little lower. This can be due to two 
causes: the changes in the courses have not resulted in improved pedagogy and thus improved results/learning; or, 
the teachers have become more strict in their grading. 

The feedback chain is reported to provide great impulse for personal and professional growth, and it 
increases the quality of the reflection papers (more depth, more content). 

Students usually indicate to have clear preferences for specific modes of reflection, which also always 
shows in the quality of their papers following that specific mode. Generally, they show least affinity with the point 
reflection. 
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Part II: Design Research Activity 



This section is used to show how the systematic way of working and investigating the context has helped to shape 
the course, and it will provide data about the quality of the course. First, some general information is presented, 
regarding the number of students attending the courses over years. Next, all resources and types of data gathered are 
described briefly. Finally, in more length, the results of the in-depth evaluation are summarized. 

General Information 

Table 4 shows the number of students that enrolled class and how many of them passed or dropped out. Overall, 
only in a very few cases were students judged an unsatisfactory mark and obliged to attend the course next year. In 
each year, there was a small number of students who dropped out. They could not cope with the deadlines for the 
sub-papers and indicated to have too little time to do it right. They enrolled in the next year. 



Table 4: # of students attending and passing the course over years (source: teacher note s) 



course cycle 


Cohort 


# enrolled students 


# of total passes 


# of dropouts 


# of failure 


grades 


6 


7 


8 


9 


1 


2002-2003 (group 1) 


10 


10 


0 


0 


2 


3 


3 


2 


2 


2002-2003 (group 2) 


22 


20 


1 


0 


5 


9 


6 


1 


3-4 


2003-2004 


56 


47 


9 


0 


5 


23 


16 


3 


5 


2004-2005 


26 


23 


3 


0 


4 


11 


8 


0 


6 


2005-2006 


22 


19 


2 


1 


4 


9 


6 


0 


7 


2006-2007 


22 


19 


3 


0 


4 


8 


6 


1 



Notes: 

• In 2002-2003 the course was delivered twice in the same period to two different groups, because of planning problems for 
students. In group 1, we focused on the role of researcher, in group 2 on the role of designer. The reflection approach was the 
same. 

• In 2003-2004 the course was delivered twice a year, due to curriculum changes. The content of both courses was the same. 
Therefore, in this table, the students are treated as one group; 

• The evaluation results from course 2006-2007 have not been analyzed yet. 

Resources 

Over years, we have systematically documented and reflected upon the process and its outcomes, in order to support 
retrospective analysis (see table 5). As such, for each course we rely on the following documents: course syllabi, 
author(s)’ articles, electronic learning system, teacher planning sheets and log files, e-mails between teachers and 
between teachers and students, course evaluations by students, student papers, written feedback on student papers by 
teachers and students, student grades, and external audit statements on the quality of the course. These sources 
reflect four aspects: 

• Our theoretical assumptions about reflection 

• Our theoretical assumptions about teaching reflection 

• Description of the process and implementation 

• Description of evaluation activities regarding the quality of the course, their outcomes and the actions taken 
according to them. 
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Course Evaluations By Students 

The University makes use of an evaluation ‘smile sheet’. Usually, the response on this evaluation is low (in the 
range between 25-50%). Also, the questions are very general, and do not provide the information we need for this 
particular course. Therefore, we have formulated an extra in-depth evaluation and hand it over during the last oral 
session. During this meeting, the student commits himself to filling in the form in return to getting the written proof 
of the grading. The response rate is between 90-100%. The results to the most important questions are presented 
below. 

Table 5: Overview of resources 





Syllabus 


Articles 


ELO 


Planning sheet 


Teach, log files 


E-mails 


‘smile sheets’ 


In-depth eval. 


Student fb 


Teacher fb 


Student papers 


Grades 


External audit 


Assumptions about reflection 


X 


X 
























Assumptions about pedagogical approach 




X 


X 


X 


X 


X 
















Process of implementation 










X 


X 






X 


X 


X 






Evaluation outcomes and actions 














X 


X 








X 


X 



What re flection approach does fit yon the best and what the worst? 

Generally, the line and triangle reflection are valued more than the point reflection or the circle reflection. 
Preferences for the line reflection relate to: structured technique; discussion of ones own norms was supposed to be 
very useful. Preference for the triangle reflection relates to the fact that more than one frame of reference needs to be 
used. This enlarges ones thinking. Overall, it is remarkable that students show an inclination for those approaches 
that were taught in the most structured way (in their eyes). Also, a positive learning experience on an approach 
connects to appreciation of that approach. 

Reflection needs to impact you as an Educational Designer. To what extent has this goal been reached? 

Overall, almost all students (>90% each year) indicated that the course has contributed to them, for a range of 
reasons, such as ‘I now know how to develop myself as a designer’, ‘I realized that different points of view is not 
necessarily negative, but that it can also serve as an inspiration for my creativity’. ‘The course has changed my 
negative attitude towards reflection into a positive one’. 1 now know how useful it is to reflect. ‘It was useful to get 
to know my strengths and my frame of reference. 

The few students who were indecisive indicated that they are not sure whether they will make use of such techniques 
often, since they found it difficult. 

The pedagogical chain in the course was: introduction to reflection, practicing reflection through exercises, write a 
paper independently -personal and general feedback. How have you experienced this pedagogical chain? 

Each year, up to 90 % of the students found the pedagogical chain effective. Most negative were the students from 
the 2004-2005 (25%), because of absence of teacher feedback. In previous and latter courses, the presence of teacher 
feedback was valued very positively. From 2005, students indicated to value the student feedback as well. 

The students were very eager to indicate potential improvements/changes. These relate to assignments to be 
formulated more clearly, the need of better step-by-step approaches of the reflection modes, strengthening the 
teacher feedback, and instruction to students on providing feedback. These suggestions have actually been 
effectuated. 

What are your experiences with working with sub-papers, building up to a final paper? 

Overall, students are positive about working with sub-papers, which culminate in the final paper. As reasons, they 
mention that it provides structure; it involves you in a process in which you get time to adjust your views, based on 
feedback; it is a good way to stay involved in the content throughout the process; it reduces the pressure at the end to 
deliver a large final paper that should be ok in one time. 

As difficult aspects, students refer to time pressure around deadlines for sub-papers. Some of them think 
adapting the sub-papers was not useful, producing a sub-paper was informative enough. 
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What are your experiences with giving and receiving feedback from peer students? 

Overall, students find peer feedback very useful, also in 2004-2005 when it was introduced for the first time. As a 
negative aspect students indicated that not every student did well on providing feedback, so that a) it should become 
part of the course goals and thus rewarded or sanctioned; b) students should be instructed better on how to provide 
feedback. This change was implemented from 2005-2006 onwards. In 2005-2006, the evaluation showed that 
students liked the combination of peer feedback, individual teacher feedback and the general feedback sessions. 



What is your opinion about the usefulness of the final oral meeting? 

For more than 90% the final oral meeting was useful. Most reported reasons are: it provided yet another step 
towards my professional growth; if have again yielded new insights; it was a nice and personal way to end a course; 
it is a good way to be able to justify to teachers the choices you have made. 

Discussion 

In this paper, we have described a way to teach reflection that differs from a lot of other approaches used in 
education (e.g. Boud & Walker, 1998; Boud, Keogh & Walker, 1985; Korthagen & Vasalos, 2002). The design and 
implementation of the course was guided by the question of ‘How to help students in the field of instructional design 
and educational technology to develop their professional expertise through systematic reflection?’ 

Firstly, Course evaluations of students show that they think the course is very relevant for their education 
and that they highly value the course. The analyses as well as our experiences provide several points for discussion 
that are related to the concept and process of reflection, as well as to the pedagogical approach. 

First, being able to ask the right questions is a crucial condition for a good reflection. This is, however, not 
easy. Students can easily stay ‘stuck’ in their own thinking. It requires that students can be loosened from their own 
thinking and their own project. Asking a second -trusted- person to ask questions from a different perspective can, 
therefore, be fruitful. It is very helpful to realize that it is important to take a point of view outside the experience, 
instead of a point of view in the experience. The former yields more information and asks for a different look at your 
project. It is very important, therefore, to choose the ‘right’ person as a reflection partner. Someone who knows the 
project well, may be inclined to ask questions from inside the project. The same holds if you act as your own 
reflection partner. Reflecting with someone who does not know the ins and outs of the project may lead to more 
surprising discoveries. In this respect, the use of ‘how’ and ‘what’-questions is more fruitful than ‘why’-questions. 
The choice out of many perspectives can make students insecure. Here is also a task for the reflection partner: to 
make people emotionally more secure. 

Secondly, reflection according to the logic of discovery is most fruitful. In this paper we described 
reflection approaches according to the logic of discovery rather than of improvement. Several students who had 
former experience with reflection approaches according to the logic of improvement started the course with a rather 
reluctant attitude. During the course, they changed their reluctance into an eagerness to learn more, and several of 
them described in their evaluation that this way of reflection was very stimulating. Such and other reactions 
strengthen our believe that the logic of discovery in this reflection approach is more productive than approaches 
according to the logic of improvement. It would be interesting to find out how teachers who are used to teach 
according to the logic of improvement, would value this way of reflection. Can their criteria be applied to this way 
of reflection, or are they two fundamentally different ways of reflecting, with own criteria and own levels of quality? 

As a third point for discussion, we argue that this reflection approach can be broadened to all professions. 
Reflection without experience is empty, experience without reflection is blind. The approaches in this paper help 
students to look at their experiences in new, different ways. By looking from new perspectives, asking the right 
questions, students can make new discoveries with respect to their profession and their own performance. The 
approaches as such are heuristics for reflection, and therefore, they are not bound to educational designers. In fact, 
we are convinced that each professional could benefit from this approach. In the education of educational designers, 

Fourthly, the depth of reflection seems to depend on the intellectuality of that person. Connecting reflection 
to Kant’s concept of judgment, means that it is related to a (personal) power to determine which concepts and 
theories are and are not appropriate for ‘concrete’ situations. Judgment, thus, is not performing homogeneous 
(logical) operations but connecting heterogeneous (logical, theoretical, personal, empirical, and practical) elements 
(Procee, 2006a). This line of reasoning means that the depth of reflection, thus, seems to depend partly on the 
intellectuality of the person. Judgment, then, is a peculiar talent which can be practiced only, and cannot be taught 
from a zero-level. Anyway, we have experienced a major growth with some students and less growth with other 
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students. Some remain having difficulty with the reflective approach. Although it is not more than a speculation, we 
got the impression that it is primarily the case with students who achieve, overall, in the whole curriculum, less than 
other students (e.g. they need more chances to pass exams, and/or they get lower grades). It would be interesting to 
see to what extent this speculation holds. First of all it needs to be determined whether those courses call upon 
judgment or understanding, and how the students score. Then it could be investigated whether there is a link 
between the results of students on those courses and on the reflection course. 

As a fifth point, we argue that the pedagogical approach should be broadened to other courses in the 
curriculum. In line with the previous remark, we would argue that the basic pedagogical approach as we apply in our 
course, should be extended to other courses in the curriculum, at least to those where is called upon judgment of the 
students. This, because this approach motivates and supports students to develop as independent professionals, being 
responsible for their own development. Further elaborating on this line, we should mention that -even for us who 
really believed in the concept- it is really difficult to take on the role of a coach, rather than of a traditional teacher. 

It may not merely be expected that all teachers are ready to take on this role. 
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Development of A Responsive Learning Environment Based on Handheld 

Devices 

Yu-Wei Wang, 

Chi-Wei Lee 
Yuan Ze University 

Handheld devices such as cellular phones and personal digital assistants (PDA) are becoming indispensable 
to our daily lives for the fast technological progress. With the convenience and mobility of handheld devices, it is 
undoubted that they will play an important role in education and learning. 

In the traditional scenario of learning, teachers usually provide learning materials and references in class 
for supplemental study. Evaluation is performed by writing down answers of questions listed on an examination 
paper. It takes time for the teacher to grade the examination papers, gather statistics of the scores and then give the 
scores back to students. With the assistance of computers, some of the steps in the above process are computerized 
and are more efficient than the traditional method. However, in most cases the evaluation and statistics can not be 
performed instantly in class. The teacher has to suspend the modification of learning until next class. 

With the ability of computation and transmitting data remotely, portable handheld devices are suitable 
substitutes for desktop computers in e-learning. In our research, we propose a learning scenario supported by 
handheld devices that let the teach can share learning materials, giving examinations, make evaluations and statistics 
and giving feedback to the students. The equipment required in our prototype is a handheld device (cellular phone or 
PDA) with the ability of Internet connection and an embedded camera, which will be easily available and reasonable 
affordable in the near future. With a downloadable or built-in program, the handheld device can take a picture of a 
two-dimensional barcode and convert it into a sequence of characters or data. 

To share digital learning references, the teacher can provide the Internet addresses of reference data to the 
students. The student can access the data with his/her handheld device immediately in class. The addresses of 
reference data can be stored in the handheld device for future study. By wireless transmission, the data can be shared 
with other students for cooperative learning. In practical situations, to key in the Internet addresses with numeric 
keys on a handheld device is a cumbersome work. We incorporate the learning scenario with a two-dimensional 
barcode reader program. The teacher can print the Internet address of reference data on a sheet in two-dimensional 
barcode format which can be shown in class. Instead on keying in the address, the students take picture of the 
barcode image and the barcode reader program will convert the barcode into the corresponding address 
automatically. 

To make an examination in class, the teach creates a webpage of online examination and store it on 
Internet. The Internet address of the online examination is also printed out as a two-dimensional barcode. The 
students then access the examination webpage with handheld device and answer questions online immediately. The 
score is calculated automatically by the server and sent to the student. The student can understand what he/she learns 
in the class. The teacher then obtains the statistics of all students’ scores right after the examination and makes 
evaluation. Due to the immediate response of the learning system, the teacher can decide to adjust the learning 
process dynamically. Since the examinations are tested digitally, the scores and related information such as dates, 
correct ratio, subjects not good at, etc. can be recorded for future estimation for both the teacher and the student. 

In general, we propose a learning scenario using handheld devices as learning assistant. The scenario is 
designed and developed to a prototype system. Combining the functions of mobile data transmission, online 
webpage, mobile webpage, database server and two-dimensional barcode, the prototype system provides the teacher 
as well as students a convenient and personalized learning environment. The immediate feedback of the handheld- 
devices-based system differs from traditional desktop-computer-based and paper-based learning environment in that 
the teacher and students can teach and learn in a responsive and diversified way. With the advantages of learning 
environment based on handheld devices, we expect that the teachers are able to create more attractive activities in 
class and the students have stronger motivation to participate in personal and group learning. 
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Implications of the Flat World for Evaluation 
in Instructional Design and Technology 

David D. Williams 
Brigham Young University 



Abstract 

This article and an associated AECT presentation explore several implications of technology innovations 
for instructional design and technology evaluation. Questions raised about changes in society associated with 
technology, based on a reading of Friedman’s World is Flat, will be discussed by participants. It is expected that 
they will be able to use what is discussed to build evaluation more systematically into instructional design and 
technology projects and processes. 



Introduction 

In his book, The World Is Flat, (Friedman, 2006) reviews several technological innovations that have 
impacted political, economic, social, cultural, and many other dimensions of the world we live in. He explores 
implications of these changes for businesses, governments, countries, disciplines such as education, and individuals 
on various fronts. Many of these implications raise questions about the role of technology in instructional design and 
associated evaluations of quality. 

After identifying several potential implications from Friedman’s work, this article explores three groups of 
questions raised and considers their value for guiding research projects and identifying alternative modes of 
designing and evaluating instruction. During the AECT session some of these questions will be presented to 
participants and they will be invited to discuss possible responses and to identify other questions they would like to 
answer through subsequent research and practice. Implications will be explored for building evaluation more 
systematically into instructional design and technology projects and processes. 

This AECT session will be highly interactive and participants will be invited to think aloud with the 
presenter to answer the questions raised, raise other questions, and begin to articulate ways to answer these questions 
in terms of their own instructional design and technology projects and processes. 

Implications from Friedman’s Flat World 

When reading through Friedman’s book, I folded down scores of pages where implications for evaluation 
in instructional design and technology came to mind. Summaries of several of the points on those pages illustrate 
how thinking about a technologically flattened world raises questions worth exploring for the improvement of 
evaluation, which might also lead to improving instructional design and the use of technology to enhance learning. 
Several such summaries follow. 

Open-Source Evaluation 

In describing the open-source invention of the Apache Web server architecture, Friedman points out in 
several ways that the participants valued evaluation of their freely offered work by trusted experts provided quickly 
and for free. The open-source context provided this atmosphere and Apache grew so strong and so fast that IBM 
acknowledged the quality of their work and asked to join their community. They were allowed to do so but “the one 
thing the Apache demanded in return for their collaboration with IBM was that IBM assign its best engineers to join 
the Apache open-source group and contribute, like everyone else, for free” (p. 103). This late 1990’s development 
points out the power of open-source collaboration which utilizes peer evaluation delivered quickly by multiple 
experts who value giving and receiving feedback as a way to improve common projects that have implications for 
proprietary projects as well. 

Friedman notes (p. 112) that the principle of “tap[ping] the innovative power of the community” can be and 
is being applied by many others in addition to software engineers. Examples include inviting the community 
scientists, engineers, and geologists to help a gold mining company find ore, involving citizens in evaluating 
community problems and giving feedback to their political leaders, and opening the newsroom and judgment of 
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news-worthiness of issues through blogging. Friedman summarizes the point this way, “The new model in business 
[as well as other areas] is that you involve your community and customers in an ongoing conversation about every 
aspect of your business, from the moment you conceive a product, to how you design it, to the supply chain that 
builds it and delivers it, to the way you collect and absorb customer feedback and respond more quickly to changing 
tastes” (p. 116). As with the other stories from Friedman’s “Brief Flistory of the Twenty-First Century,” what are the 
implications for how evaluation is conceived as part of the instructional design process? What could be done to use 
open-source communities to provide rapid and expert feedback to teachers, learners and people trying to support 
them? 

Technological Feedback Immediacy 

Using Wal-Mart as a key example, Friedman points out several ways technology has been used to speed up 
feedback to employees and suppliers so they can more easily and cheaply provide their services, driving down the 
price of the supply chain, and ultimately the products customers want. Fie describes (on pages 159-162) the use of 
radios and satellites to guide drivers so they rarely drive an empty truck, computerized instructions and rate 
feedback through headphones to pallet movers, opening sales and inventory databases to suppliers to enhance a just- 
in-time inventory program in which suppliers are viewed as partners, and the use of radio frequency identification 
microchips to replace barcodes which allows “Wal-Mart to track any pallet or box at each stage in its supply chain 
and know exactly what product from which manufacturer is inside, with what expiration date” (p. 161). These 
innovations provide evaluative feedback to people who need it to produce and ship products customers want and 
will buy. Flow would instructional design be different if similar technologies were used to enhance the speed and 
quality of evaluative feedback from learners, teachers, and other stakeholders designers want to serve? 

Insourcing Means Trusting Your Evaluator 

Using the United Postal Service (UPS) as a model, Friedman notes that by building trust through basic 
delivery services over years, this company has been allowed “deep inside” several companies, large and small, to 
evaluate what is wrong with supply-chains. They then provide recommendations and services that save the 
companies money and even take over many of these functions so the companies they serve can focus on only a few 
of their original business tasks. UPS has used wireless technology to enhance accurate and efficient delivery and the 
Internet to encourage customers to track their own packages. They have found ways to use technology to allow them 
to collaborate more intimately and extensively than ever before. “In many cases today, UPS and its employees are so 
deep inside their clients’ infrastructure that it is almost impossible to determine where one stops and the other starts. 
The UPS people are not just synchronizing your packages — they are synchronizing your whole company and its 
interaction with both customers and suppliers” (p. 175). Is that level of trust possible in instructional design? Will 
teachers and learners trust instructional designers “deep inside” their learning experiences and are designers 
prepared to evaluate the needs and experiences of learners and teachers in sensitive but accurate and rapid ways so 
their “customers” can see that they have provided them with information they didn’t have without this help? Are 
there ways technology could help designers do this? 

Everyone Can Be Their Own Evaluator 

Using the Internet, wireless services, TiVo, Google, digitization, and other technological innovations, 
Friedman points out that more and more people are able to access and explore incredible amounts of information as 
they choose to do so and they are also able to evaluate programs, products, and people associated with that 
information using whatever criteria they personally want to employ. In other words, everyone is an evaluator and 
everyone and everything they do or produce is more easily evaluable by them and everyone else. As Friedman 
summarizes, “In a flat world, you can’t run, you can’t hide, and smaller and smaller rocks are turned over. Live your 
life honestly, because whatever you do, whatever mistakes you make, will be searchable one day” (p. 185). What are 
the implications for learners and teachers? Should instructional designers be helping learners evaluate themselves 
and all the potential aids to their learning more efficiently and effectively through use of these technologies? Could 
designers be evaluating their design efforts formatively and through the evaluative eyes of their clients better if they 
took advantage of these technologies? Whether they do or not, will learners, teachers, and other clients find ways to 
make those evaluations themselves? 

These examples and several others in Friedman’s book, various newsletters and technology updates remind 
me that evaluation is a central part of what we do as humans and technology can help us perform our decision 
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making, information processing, moral judging better if we will allow it to. If we don’t, we are likely to be judged 
and evaluated by others who have this same power, before we can make the adjustments we want to make. 



Three Groups of Questions 

Three groups of questions raised by a reading of The World is Flat and their potential value for guiding 
research and adjusting modes of instructional design and evaluation are presented below. These questions along with 
those raised earlier will be used to stimulate discussion during the AECT convention session. Participants will be 
invited to identify possible responses and additional questions they would like to address through future research 
and practice. 

Agency 



Given that technology is making the world more accessible to individuals, their evaluation skills, 
responsibilities, and powers are growing as well. In a business sense, technology is helping the customer, who wants 
to, become better informed and more discriminating or selective in what they consume. Likewise, in terms of 
learning, taking advantage of the privileges available through so many sources to anyone with access to the Internet 
and the world it opens is becoming easier for individuals everywhere. Because more people have more options or 
choices, they are more and more free to choose what they want to learn, how they want to learn it, and how to know 
if they have learned enough. In short, technology is making it possible for more of us to be full agents over our own 
learning. 

With this power comes a huge responsibility as well. More and more, learners can be viewed as responsible 
for their own learning and not so dependent on teachers and others to guide them. In terms of evaluation, this means 
that individuals are more clearly responsible for evaluating potential information, experiences, learning 
opportunities, and teachers. They are also more responsible, therefore, to clarify what their own values are, what 
criteria they will use in judging potential learning experiences, to gather evaluative information about various 
resources, and to make their own evaluative judgments and decisions they will live by. 

But to really have evaluation agency, individuals have to be aware that they have this responsibility. They 
need to develop skills of critical thinking and bias control. They need to become disciplined in their evaluation skills 
if they are to fully realize their evaluation powers and appropriately use their evaluation responsibilities. Simply 
having access to tools doesn’t guarantee people can appropriately use those tools. 

These realities about human agencies have huge implications for learners, teachers, and instructional 
designers who want to assist them. Several questions we ought to explore in understanding and dealing with these 
implications are listed below. 

Questions Raised 

Are we recognizing the role of agency and how technological innovations are impacting learners’ agency? 
How are instructional design and evaluation theories and approaches taking this individual agency into account, if at 
all? Are evaluations associated with instructional design projects taking agency into account? How? How might they 
do this better? How are instructional technologists addressing issues of responsibility, self-interest, self-confidence, 
and the balance between individuals and the groups/organizations/collaboratives they are part of? What are we doing 
to develop theories and practices in instructional design, technology, and evaluation that respect and build upon the 
agency of the learners and teachers we serve? 

Value for Guiding Research, Design, and Evaluation 

Is it important to answer these questions about agency? Based on my reading of Freidman book and many 
other sources, taking human agency into account is one of the most important things we should be doing as we 
pursue the learning sciences, learning theories, instructional design practices, and evaluation of interventions. If we 
do not take into account who the people are that we want to encourage to learn better and if we do not build upon 
their responsibilities as well as their access and other powers that are enhanced by technological progress, we are 
morally responsible for violating or at least ignoring that agency. We will spend time in this session exploring that 
possibility and what we are doing with OUR agency to respect the agency of learners. 
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Sharing Values 

As Friedman has documented, advances in technology have made it easier for everyone who has access to 
them to be exposed to a wider and wider variety of ideas and values. Living in a world flattened by technologies 
such as the Internet, cell phones, satellite television, etc. potentially exposes many more people than ever before to 
other cultures, other lifestyles, other religious views, other experiences, and other values. 

This exposure makes it possible for evaluators and all the people they serve (designers, clients, learners, 
teachers, and so on) to better understand others’ values and perhaps to reconsider their own views. However, people 
may still resist understanding others’ values for a variety of reasons because, as mentioned earlier, they are agents 
and can choose to attend to whatever values they want to attend to. 

A key issue in all evaluation studies is the role of stakeholder values. For many years only selected 
stakeholders and their values have guided many evaluations. For example, in schools, tests have traditionally been 
built by teachers to reflect the parts of the curriculum they valued most and felt students should also value. In 
businesses, the values of business owners and leaders usually take precedence over the values of the workers if they 
are at odds. In instructional design, the values of the people sponsoring the development of an instructional product 
and the values of the designer often take precedence over the values of the consumers or learners or the people they 
might serve with whatever they learn from the product. 

However, in recent years evaluation theorists, such as Patton (1997) and Cousins & Earl (1995) have 
pointed out convincingly that for evaluations to be truly useful and worth the effort, all the stakeholders who are 
impacted by the thing being evaluated and thus, by the evaluation, should have a role to play in voicing their values 
and having their perspectives addressed in evaluations with which they are associated. In other words, the field of 
evaluation has followed the trends illustrated by Friedman and seeks to account more equitably for the multiple and 
often conflicting values of all the stakeholders involved. 

Technology might make this process easier but it certainly makes it clear that there are many more values 
to be taken into account than evaluations used to attend to. Having more ready access to multiple value perspectives 
provides a challenge to traditions that may impact instructional designers as well. How they use evaluation to 
address those values with or without using technological innovations is an issue that ought to be addressed by 
theoreticians as well as practitioners. 

Questions Raised 

How does technology help reveal the perspectives of multiple stakeholders? Is it used to conceal or ignore 
certain values as well? How do instructional design and evaluation theories take multiple values into account? How 
do they ignore or inappropriately balance alternative perspectives? How are stakeholders’ views and values shared 
and built into evaluations and associated instructional design projects? How could they be better shared? Could 
using technology help with this task? What should we be doing about the use of technology to understand and share 
values to enhance instructional design and its evaluation? 

Value for Guiding Research, Design, and Evaluation 

The lessons learned by evaluation experts indicate that if all the stakeholders’ views are not acknowledged, 
the evaluation results are less important to many stakeholders. They are less likely to value evaluation conclusions 
and more likely to ignore them instead of using them to improve their practices. This is likely the case for 
instructional design as well. If the learners’ values are not included in formative and summative evaluations of an 
instructional product, why would they value the product or conclusions about it? Thus, for practical reasons, as well 
as moral imperatives, the people who have an interest in or who are impacted by an instructional process or product 
should also have their values included in the evaluation and shaping of that instruction. Research into these issues 
must take the participants’ views and values into account or it is not worth doing. 

Trust Issues 

The example cited earlier of UPS using technology to get deep inside various organizations to evaluate 
their financial, supply-chain, and other processes raises several trust issues. In contrast the many examples given by 
Friedman regarding how much information is available on the Internet regarding organizations and individuals 
suggest some different but related issues. In all the cases Friedman explores, it is clear that people can and will use 
technology to gather and interpret information that may be considered private or sensitive by others. However, often 



363 




people are willing to share that information if they believe they can trust others to use that information for their 
mutual benefit. In contrast, many individuals may feel powerless to protect information about themselves and thus it 
is possible they will develop distrust of various technologies and the people using them. Implications for 
instructional designers who use technology and for evaluators who seek to help instructional designers improve their 
practices might include the following: 

1 . Designers cannot simply assume that the clients or consumers of their products will use technology 
built into the designs. Although various technologies may provide the most efficient means for sharing 
the products, they may not be the most effective if the users distrust and refuse to use them. 

2. To get to know the clients and consumers and their needs, designers may be tempted to use 
technologies that these people may not trust. Sensitivity in how needs assessment information is 
gathered and interpreted is essential. 

3. Designers and evaluators who are helping them may also want to use technologies or assume the utility 
of technologies in evaluating the implementation and impact of various products. Exploring how 
stakeholders view the technologies should be part of the evaluation process. 

4. Clients and others to be served by designers and evaluators may be using technologies to gather their 
own evaluative information about the designers and evaluators in more sophisticated ways than 
expected. Trust goes both ways and all parties’ trustworthiness is more easily gauged through evolving 
technological sources. 

These trust issues highlight another growing movement in the professional evaluation field that might be 
important to consider by both instructional designers and evaluators. Promoted by David Fetterman (2001), 
empowerment evaluation has been evolving as a practiced approach for the last decade or so. This approach to 
evaluation involves evaluation experts serving as coaches to guide, encourage, and train various stakeholders 
(usually those who feel they have been in some way disenfranchised as stakeholders by an organization, group, or 
culture in which they function) to do their own evaluations and to use evaluation as a tool to improve their situations 
socio-economically, politically, and/or in other ways. Possibly, evaluation from this perspective is based on a 
distrust among some stakeholders that their values are going to be represented well by traditional evaluators. If 
instructional designers were to work with clients who wanted to use empowerment evaluation of their instructional 
products, they too would want to be sensitive to the trust issues involved. Finally, empowerment evaluation 
stakeholders may be particularly interested in using technologies that would allow them to gain the power necessary 
to conduct their own social agenda-oriented evaluations. 

This example of empowerment evaluation and the trust issues associated with technology and other 
influences suggest the importance of examining the basic process that evaluations usually follow to see what other 
implications there may be for trust issues associated with design and evaluation practices that involve various 
technologies. The evaluation process is rather straightforward, though often politically charged and includes the 
following steps as summarized by Williams (2006): 

1. Clarify who cares about the thing to be evaluated; who are the stakeholders and who’s values should 
be included (e.g., are the consumers’ and the funders’ views equally important)? 

2. Help the stakeholders clarify what they care about; what is the evaluand or thing to be evaluated and 
what aspects of it are they most interested in (e.g., an instructional product)? 

3. Help the stakeholders clarify what their values are and what criteria and standards or levels of 
performance on the criteria those values imply forjudging the quality of the evaluand (e.g., how much 
and what learners should learn and at what level of performance when using the instructional product). 

4. Use steps 1-3 to identify the evaluation questions stakeholders want answered. 

5. Develop data collection, analysis, and reporting strategies to answer the questions, focusing primarily 
on the main question, which is “how well does the evaluand meet the specified criteria?” 

Reflection on this process reveals the point that stakeholders and their values guide the entire process. 
Therefore, trust levels between stakeholders and evaluators and among stakeholders can greatly impact the 
identification of criteria, definitions of the evaluands, questions to be asked, and methods for conducting the data 
gathering, interpretation, and reporting. If use of technologies by the stakeholders, by the instructional designers of 
the evaluands being evaluated, or by the evaluators introduce trust or reduce trust among the stakeholders at any of 
these stages, the entire enterprise will be affected. 

Questions Raised 

Is something like what UPS is doing desirable for design organizations? What would it take to do this? 
What could the role of technology be in doing this? Will instructional design and technology organizations allow the 
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kind of deep involvement by evaluators that businesses have shown with UPS as their evaluator? What are the trust 
issues involved and how can trust be built? How do stakeholders use technology to do their own evaluations of their 
own operations and of potential instructional designers and/or evaluators? How does the individual empowerment of 
stakeholders through the use of technologies such as Internet search engines and/or the use of empowerment 
evaluation approaches impact their trust of professional instructional designers and evaluators and vice versa? How 
is the evaluation process being used in instructional design projects that use some of the new technologies? Could 
that process be more effective if it were refined? If so, how? What would be the interaction between a refined 
process, use of technologies, and trust? 

Value for Guiding Research, Design, and Evaluation 

Are the issues discussed earlier regarding trust, technology, and various approaches to evaluation important 
for guiding research on and the practice of instructional design and evaluation? The relationships among the various 
approaches to evaluation and how they are applied to instructional design are complex. They are further complicated 
when designers use technological innovations that involve the trust issues described earlier. Likewise, even if 
designers use more traditional tools, many of their consumers and clients may be using technologies that empower 
them as individuals and organizations. And these uses of technologies may complicate the trust needed between 
designers, clients, consumers, and evaluators in ways not well understood. Research into this phenomenon would be 
worth pursuing. 

Conclusions and Implications 

Although many other ideas will likely be stimulated by the discussion during AECT, tentative conclusions 
and implications of this analysis are summarized below. 

First, the fields of instructional design and evaluation have always been related; but technological 
innovations and their influences on both fields could stimulate more thoughtful attention to how these related fields 
could be more integrated. We should examine more thoroughly how instructional designers can and could use 
formal internal and external evaluation processes to assess needs, weigh the strengths and weaknesses of alternative 
ways to meet identified needs, formatively evaluate emerging designs and development efforts, evaluate 
implementation fidelity of developed programs and products, as well as summatively judge the ultimate 
effectiveness of these programs and products in achieving their objectives. 

Likewise, how are the various approaches to evaluation that have been developed over the last 40 years 
being shared with and used by instructional designers to enhance their work? Do designers know about 
empowerment (Fetterman, 2001), participant (Cousins & Earl, 1995), responsive (Stake, 2003), utilization-focused 
(Patton, 1997), fourth-generation (Guba & Lincoln, 1989), and other approaches to evaluation (see Stufflebeam, 

2001 for a review of several approaches)? If they do know about them, are they using them? If so, how? If not, why? 
How could these approaches be modified to be more helpful to instructional designers and developers? 

Second, innovative ways of using and developing technology explored by Friedman come from essentially 
every discipline and field of interest in the world today. Reading about them raises questions regarding how open 
instructional designers and evaluators are to using what others are discovering from other fields AND what we are 
doing to discover and innovate new uses for technology ourselves. Are we using the lessons learned by others? Are 
we willing to explore uncharted territory as well? What are we doing to take advantage of the new tools being 
shared by people throughout the world? What conversations are we having with people in various fields that would 
allow us to be stimulated in our creativity and to help them help us develop our own tools? 

Third, agency, sharing, and trust are key issues raised by this analysis of the flat world. How are we dealing 
with these issues in our research and practices associated with instructional design and evaluation? Should we be 
addressing these issues? What difference would it make if we did? The analyses in this article conclude that these 
are important concepts that have powerful implications for work we are doing and could do. How can we encourage 
researchers and practitioners alike to take these issues seriously? How do instructional designers and evaluators 
think about people (one another and their clients and consumers) in terms of their agency, in terms of their 
trustworthiness and trustfulness, and/or in terms of their willingness and ability to understand and share their values? 
On the other hand, how do clients and consumers of design and evaluation projects think about each other and their 
project designers and evaluators in terms of agency, trust, and sharing? Does it matter how these questions are 
answered? If so, what can be done about obtaining answers and what can be done with results? What should we be 
doing as members of AECT and as an organization to address these issues? 
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Fourth, are the implications raised in this article from a reading of Friedman’s book worth further 
consideration? This article claims that they should be explored further and perhaps instructional designers and 
evaluators should experiment with these ideas in their research and practices. In particular, lessons learned from 
open-source communities could be synthesized. Instructional designers and evaluators could experiment with using 
open-source procedures for enhancing both the design of instruction and the evaluation of those design efforts. 

Similarly, could the immediacy of feedback available through new technologies that various industries 
have begun to use to continually inform their performance help instructional designers and evaluators in educational 
settings too? Although people and their learning are very different from products sold at Wal-Mart, are there 
practical ways to use the electronic feedback and evaluation systems Wal-Mart uses in instructional designs, 
schools, and other learning settings? Creatively exploring ethical and humane ways of using technology to monitor 
student interest, learning, performance, attention, and so on seems like a powerful way for AECT members to 
transfer lessons learned from others to their settings. Use of student feedback clickers (such as TurningPoint) is one 
example that some innovators have been exploring. What others might we consider if we took the experiences of 
entities such as Wal-Mart seriously in our own contexts? 

Trust-building between evaluators, instructional designers, and those they serve is another issue that was 
addressed somewhat in this article. What else should be done to explore the role of trust in the design process? How 
would viewing each other as agents modify our views of and approaches to trust-building? How do stakeholders 
view evaluation and evaluators and how does that affect trust? What could be done using technology innovations to 
overcome distrust? How can instructional designers and evaluators come to see that trust-building is essential and 
not something that can be accomplished simply through typical evaluation methods? 

Finally, are instructional designers and evaluators aware of the power individual clients and consumers are 
gaining through access to information newly available by use of innovative technologies? Do they see how that 
power can be used effectively to assess consumer needs, design solutions, deliver instructional services and 
products, and evaluate the entire process and outcomes? Do they see the interface of participants’ agency, trust, and 
power with the instructional design and evaluation objectives? What differences could be achieved in learning and 
growth if these issues were more explicitly addressed by professionals represented by AECT? 

This AECT session will provide an opportunity for participants to explore a few of these many questions. 
However, you are invited to join in this conversation as well by contacting me and sharing your thoughts, 
suggestions, and additional questions. Please contact me at david_williams@byit.edu. 
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Abstract 

Research shows that asynchronous online discussions, if appropriately implemented, can increase student 
knowledge and understanding of course materials (Garrison, Anderson, & Archer, 2001). However, not all students 
are partial to learning in an online environment. Past studies indicate that individual learning preference is one major 
factor that prevents some students from participating in and posting in online discussion forums. This study 
investigated 1) online students’ learning preferences/learning styles and interaction styles in asynchronous online 
discussions with and without instructor’s presence in the asynchronous online discussions, 2) the relationship 
between online students’ learning styles and interaction styles; and 3) factors impacting students’ interaction styles. 
This study has implications for the design and organization of effective asynchronous online discussions and the 
quality of online learning. It also contributes to effective strategies helping students adopt a more active interaction 
style and achieve better learning outcomes in online learning environments. 

Introduction 

The number of online courses and the number of students enrolled in them continues to grow; as of 2005, 
3.2 million students were enrolled in online courses (Allen & Seaman, 2006). Within online courses one of the 
major pedagogical practices is the use of asynchronous online discussion (McLoughlin & Luca, 2000). In 
asynchronous online discussions, students demonstrate different interaction styles, which are the ways or habits they 
acquire knowledge from online discussions (Sutton, 2001). For example, some learners are constantly participating 
or posting more than the course requires, which allows them to be categorized by Sutton's work as having an active 
interaction style (Beaudion, 2002). Some are actively observing and processing both sides of the interaction from 
others without direct interaction, which can be categorized as a vicarious interaction style (Sutton, 2001). Moreover, 
some students may be neither actively involved nor an observer, which we refer to as the mixed or balanced- 
interaction style. For students categorized within the mixed or balanced-interaction style, their level of effort 
approximately equals to the minimum amount of postings of the course requirements. 

According to Sutton (2000), all participants in asynchronous online discussion are involved in vicarious 
interaction and benefit from vicarious learning. However, the benefits of vicarious interaction “will not be as great 
as in the case of direct [interaction]” (Sutton, 2000, p.23). As such, designers and instructors of online learning 
environments should be aware of students’ different interaction styles and try to promote more direct interaction in 
asynchronous online discussions. Correspondingly, it is important to understand what affects students’ direction 
interaction in asynchronous online discussions. 

Learning Styles 

Individual learning style has been reported to be one of the main factors that prevent online students from 
directly participating in asynchronous online discussion (Beaudion, 2002). Individual learning style is the cognitive, 
affective, and psychological traits (Keefe, 1979), which students reveal when interacting with, perceiving, and 
responding to the online learning environment. Individual learning styles played a major role in the way students 
learn and process information in online environments (e.g., Assis, Danchak, & Polhemus, 2005; Ford & Chen, 2000; 
Riding & Cheema, 1991). This somehow leads us to believe that learning styles are static characteristics (Assis, 
Danchak, & Polhemus, 2005; Pena, Marzo, & Rosa; 2002). With this static characteristics assumption about 
learning styles, online instructors would primarily focus on how to accommodate different learning styles and 
preferences in order to promote more direct interaction. 

However, according to Kolb (1984), individual learning styles are dynamic and change over time because 
of different elements, such as learning objectives and learners’ role in the learning process. Butler and Pinto-Zipp 
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(2005) found that there was no single dominant learning style of students in online courses and most online students 
displayed a dual learning style. Similarly, Fahy and Ally (2005) found that students’ cognitive styles were not 
significantly correlated with their preference for instructional delivery modes. Therefore, it is viable we can treat 
individual learning styles as dynamic characteristics and focus on how to direct students to adapt to and change their 
roles in online learning environments. 

The purpose of this study was to investigate online students’ learning styles and online interaction styles in 
asynchronous online discussion with and without instructor’s presence in the discussion. It also looked at the 
relationship between online students’ learning styles and their online interaction styles. Specifically the research 
questions were: 

• What kinds of interaction styles (active, vicarious, or mixed style) did the students display in the 
asynchronous online discussion with and without instructor’s presence? 

• What kind of learning styles did students display in the asynchronous online discussion? 

• What factors impacted students’ interaction styles in the asynchronous online discussion? 

Method 

The context of the study was two graduate Educational Technology courses offered at a large Midwestern 
university; twenty-one students voluntarily responded to the surveys. The first course was an introduction to 
instructional design and the second one was a foundations of distance education. Asynchronous online discussion 
was a mandatory course component without instructor’s presence in the first class, which had weekly face-to-face 
meetings. Asynchronous online discussion was the main instructional method with instructor’s participation and 
facilitation in the second class, which was delivered via WebCT with a few face-to-face meetings. 

The participants were the students who enrolled in both classes. The student’s body had a wide variety. 
There were graduate students, school administrators, and other working professionals. The participants were from a 
variety of different fields too, such as education, technology, computer graphics, and engineering. Both courses had 
students with different ethnicities, such as white (n=14), Africa American (n=3), Hispanic (n=2), and Asian (n=2). 
Students’ ages ranged from 21 to more than 40 years old. Students had different levels of experiences with online 
discussions and online courses. Thus, participants had a wide variety in terms of student characteristics in online 
courses. 

A student online interaction style survey was administered to all students enrolled in two classes at the end 
of the courses. Survey included Likert-scale items focused on students’ online interaction styles, online learning 
styles, factors impacting online interaction styles, and online learning activities. It also included some open-ended 
questions that were intended to solicit other factors impacting student interaction styles and students experiences 
with and without the instructor’s presences in online discussions. The Likert-scale items were created based on 
definitions of vicarious learning, i.e., learning through observing (Banduara, 1986) and vicarious interaction, i.e., 
observing and processing both sides of the interaction/discussions (Sutton, 2000). Some of the questionnaires that 
are tracking “lurking” students in online discussion created by Michael Beaudoin (2002) were also included. 

Sample items are: 1) I often processed ideas from reading others’ postings even when I was not visibly participating 
in the online discussion; 2) I was more of an autonomous learner and seldom got too engaged in group online 
discussion; 3)1 preferred interacting and discussing the course content materials with others in order to learn more 
effectively; 4) I would not have participated in the online discussions/postings if it was not graded; 5) I preferred 
reading others’ postings and comments to writing my own discussion postings. 

For the confirmation purpose, we triangulated the data from students’ responses of their interaction styles 
and learning styles with their learning activities and participation activities (such as number of postings and logins) 
in WebCT to classify them into different interaction and learning style categories. The survey was also pilot-tested 
and modified. 



Results 

Survey results indicated that students (N=21) employed the various interaction styles defined by Sutton's 
work in asynchronous online discussions (see Table 1). With instructor’s presence in terms of facilitating and 
monitoring the online discussion, more students (n=7) displayed active interaction style in the asynchronous online 
discussion. Without instructor’s presence in the asynchronous online discussion, more students (n=5) displayed 
mixed or balanced-interaction styles in the asynchronous online discussion. With instructor’s presence in the 
discussion, only one student (n=l) identified his or her participation as vicarious interaction. 
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Table 1: Students’ interaction styles in asynchronous online discussion 



Interaction Style 


Without instructor’s 
presence 


With instructor’s 
presence 


Total Responses 




# of 

responses 


Percent 


# of 

responses 


Percent 


# of 

responses 


Percent 


Active Interaction 


3 


30 


7 


64 


10 


48 


Vicarious Interaction 


2 


20 


1 


10 


3 


14 


Mixed Interaction 


5 


50 


3 


26 


8 


38 



In addition, students displayed different learning styles in the asynchronous online discussion (see Table 2). 
It was interesting that without instructor’s presence, 90% of the students would prefer to discuss content materials 
with others; while with instructor’s presence, 91% of the students reported that they processed ideas from reading 
others’ postings without visible participation. Most students seemed to have dual learning styles either with or 
without instructor’s presence. But with instructor’s presence in the online discussion, only 27% of the students 
indicated that they were autonomous learners and seldom got too engaged in group online discussion. 



Table 2: Students’ online learning styles in asynchronous online discussion 



Learning Style 


Without instructor’s 
presence 


With instructor’s 
presence 




# of 

responses 


% of Total 
responses to 
the item 


# of 

responses 


% of Total 
responses 
to the item 


Preferred to discuss content materials with 
others 


9 


90 


6 


55 


Preferred to read others’ postings 


6 


60 


6 


55 


Was an autonomous learner and seldom got 
too engaged in group online discussion 


6 


60 


3 


27 


Processed ideas from reading others’ 
postings without visible participation 


7 


70 


10 


91 



The main factors impacting students’ interaction styles were individual learning styles, such as preference 
for discussing content materials with others (N=16) and preference for reading others’ postings (N=12), instructors’ 
involvement in asynchronous online discussion, and course requirements (e.g., requiring peer feedback). Other 
factors such as time, class size, and content difficulty also impacted students’ interaction. However, there was no 
strong indication of a specific learning style for a particular online interaction style. For example, although students 
may have indicated particular learning styles, they could have also indicated the same interaction style. 

Discussion 

There were more active interaction style learners in the asynchronous online discussion where the 
instructor was directly involved in the discussion and students were required to respond to others’ postings. Without 
online instructor’s presence in the asynchronous online discussion, more students easily “melted” into the mixed- 
interaction styles, the “didn't do a lot and . . . didn't do a little” and “did just enough” to meet the minimum 
requirement(s). Consequently, the chances of in-depth discussions are greatly reduced. Thus, instructors’ facilitation 
and monitoring in asynchronous online discussion are critical for effective learning and positive learning 
experiences in asynchronous online discussion. 

This study also demonstrated that students displayed dual learning styles in asynchronous online 
discussion, which confirmed that learning styles are dynamic (Kolb, 1984). The result was also consistent with 
previous research findings on students’ dual learning styles in online learning environments (Fahy & Ally, 2005; 
Pinto-Zipp, 2005). However, it failed to find any direct association between learning styles and interaction styles. 
For example, although most students (at least 60%) in the online discussion without instructor’s presence indicated 
preference for active discussion, only 30% of the students were actively participated in the online discussion. On the 
other hand, only 55% of the students in the online discussion with instructor’s presence indicated preference for 
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active discussion with others, 64% of the students were of active interaction style. This finding tells us that even 
students are of “passive” learning styles, with effective guidance and feedback, they can become active learners and 
active participants in online learning environments and vice versa. 

Although learning styles or preferences were the strongest factors impacting students’ interaction styles in 
asynchronous online discussions, online instructors could influence or change students’ predominant interaction 
styles. Online instructors can help online students become more actively interact with others through effective 
facilitating and commenting on different perspectives. In addition, some students responded that effective course 
design, such as providing interesting discussion topics and sharing participants’ background information, can also 
help students’ adapt to a more active interaction style 

Although the researchers made very effort to correctly identify students’ interaction styles and learning 
styles, there was no guarantee that every student was correctly classified into the right category because several 
students reported that they have changed interaction styles/learning styles as the course preceded. Thus, future 
studies should explore why and how students change their interaction styles and learning styles in online learning 
environments. In addition, the participants were graduate students and most of them had previous asynchronous 
online discussion experience(s), which may affect their interaction styles or learning styles in the online discussion. 
Finally, the sample size is quite small. Therefore, more data from a wide variety of student body in terms of online 
learning experiences are needed. 
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Designing An Online Course: What Does It Take? 
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Abstract 

What does it take to design an online course? This paper discusses the question and highlights online 
course design challenges which can’t be easily resolved according to technology-mediated or online course design 
literature. Initiated from the first-hand knowledge of a case study, the discussion is centered on the review of online 
course design literature, online course designers’ challenges and solutions, and online students’ feedback on 
designers’ solutions for design challenges. Implications help overcome real-world design obstacles. 

Introduction 

Designing an online course is a complex and challenging task. First, online course designers can not simply 
transfer traditional instructional methods online without adaptations (Moore & Kearsley, 1996). Second, with the 
prevalence of online courses in both higher and K-12 education, more online course designers without extensive 
educational background are facing challenges in online courses development (Shrivastava, 1999). Now, we ask: 
what does it take to design an effective online course? In order to answer the above question, we’ll have to look at 
the literature and find out what the literature offers for online course designers first. 

Literature Review 

A wealthy amount of literature related to technology-mediated, especially online course design, can be 
roughly classified into three categories: 1) critical factors impacting students’ online learning; 2) models and 
frameworks for online course design; and 3) practical guidelines for online course development. 

The first critical factors category provides an overview of major factors impacting students’ learning and 
learning experiences in online environments. These factors are online interaction (e.g., Moallem, 2003; Moore, 
1989), a sense of presence (e.g., Shin, 2002; Tu, 2000), online community (e.g., Mayer, 2005; Stepich & Ertmer, 
2003), and the role of online instructors (Berge, 1995), to name a few. Such literature resources define and explain 
online interaction, presence, online community, the role of online instructors and the importance of such factors of 
effective online learning. In summary, the first category provides a knowledge base, which tells what online course 
designers should focus on and why they should focus on those factors during the design process. However, online 
course designers still have to make their own decisions regarding critical factors in their contexts. 

The second models and frameworks category provides procedures and steps in the process of online course 
design and strategies for integrating instructional principles and learning theories into the design. These models and 
frameworks usually contain a combination of a set of instructional design principles for analyzing the content and 
learners (e.g., Johnson & Arogan, 2003; Shearer, 2003) and promoting social interaction and a sense of presence 
(e.g., Moallem, 2003; Northrup, 2001). However, most models and frameworks require instructional design 
background to comprehend and implement them effectively. 

The third practical guidelines category provides general guidelines for online course development. For 
example, with the prevalence of online courses, some organizations and universities have created/adapted their own 
guidelines for online course development. In reviewing these guidelines, we found that most guidelines are too 
general to provide solutions for real design challenges. In addition, most guidelines are for the purpose of evaluating 
online courses rather than directing the online course design process. 

A Case Study 

A graduate online course was designed by an advanced doctoral student and a faculty member at a large 
Midwestern university. The course was on online education and targeted for graduate students, faculty members, 
and administrators. It was co-taught by the same two designers. At the close of the course in Fall 2006, students’ 
feedback and reflections on their learning experiences, especially on specific course design features, were solicited 
through an online survey. 
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Challenges Encountered 

During the design process, we encountered several major challenges which were unique in online course 
development and were not readily to be resolved according to the literature. The major challenges encountered were: 
1) how to select the course content to meet the most varied students in terms of entry level knowledge, given the 
potential that an online course provides opportunities for more knowledge seekers; 2) how to introduce the content 
to students effectively and efficiently rather than telling students to read everything or posting “a real long lecture” 
(e.g., PowerPoint slides or video clips); 3) how to help students process the large amount of information generated 
from weekly online discussion. 

Knowing the characteristics (e.g., information overload) of technology-mediated instruction and our unique 
context, we came up with our own design solutions, such as preparing two (introductory and advanced) levels of 
reading materials and the “something to think about,” which were one -page summaries of weekly readings that 
highlighted important ideas for weekly online discussions. Our students responded positively to these solutions, 
which saved students’ time, avoided information overload, and most importantly, directed students’ learning. 
Students’ positive feedback confirmed our design solutions and provided insights into the online course design. 

Conclusion 

According to our knowledge and findings gained from the case study, we concluded that it takes content 
(subject) knowledge, the knowledge of online education, and the knowledge of different models and frameworks for 
online course development to design an effective online course. It also takes an open and detail-oriented mindset. 
Last but not the least, online course designers should be prepared to collect and solicit online students’ feedback on 
different aspects of an online course during the design process. 

We hope that our study and the solutions we presented can help more online course designers overcome 
their challenges. We also hope there will be more such discussions that can be added to the literature to guide online 
course designers. 
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